A Differential Evolution-Enhanced Position-Transitional Approach to Latent Factor Analysis

Latent factor (LF) models are highly effective in extracting useful knowledge from High-Dimensional and Sparse (HiDS) matrices which are commonly seen in various industrial applications. An LF model usually adopts iterative optimizers, which may consume many iterations to achieve a local optima, resulting in considerable time cost. Hence, how to accelerate the training process of an LF model becomes a highly significant issue. To address it, this work innovatively proposes a randomized latent factor (RLF) model. It incorporates the principle of randomized learning techniques for neural networks into the LF analysis on HiDS matrices to alleviate the computational burden greatly. It also extends the standard learning process for randomized neural networks in context of LF analysis to make the resulting model represent an HiDS matrix correctly. Experimental results on three HiDS matrices from industrial applications demonstrate that compared with state-of-the-art LF models, RLF is able to achieve significantly higher computational efficiency and comparable prediction accuracy for missing data. More importantly, it provides a novel, effective, and efficient approach to LF analysis on HiDS matrices.

How to extract useful knowledge from a high-dimensional and sparse (HiDS) matrix efficiently is critical for many big data-related applications. A latent factor (LF) model has been widely adopted to address this problem. It commonly relies on an iterative learning algorithm like stochastic gradient descent. However, an algorithm of this kind commonly consumes many iterations to converge, resulting in considerable time cost on large-scale datasets. How to accelerate an LF model's training process without accuracy loss becomes a vital issue. To address it, this study innovatively proposes a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</u> ultilayered-and-randomized <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</u> atent <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</u> actor (MLF) model. Its main idea is two-fold: a) adopting randomized-learning to train LFs for implementing a ‘one-iteration’ training process for saving time; and 2) adopting the principle of a generally multilayered structure as in a deep forest or multilayered extreme learning machine to structure its LFs, thereby enhancing its representative learning ability. Empirical studies on six HiDS matrices from real applications demonstrate that compared with state-of-the-art LF models, an MLF model achieves significantly higher computational efficiency with satisfactory prediction accuracy. It has the potential to handle LF analysis on a large scale HiDS matrix with real-time requirements.

A latent factor analysis (LFA) model can efficiently address a high-dimensional and sparse (HiDS) matrix with a stochastic gradient descent (SGD) algorithm. However, an SGD-based LFA model's performance depends heavily on its hyper-parameters. The popular method is based on grid-search, which fails because of expensive computation and time-consuming. Aiming at implementing a hyper-parameter-free LFA model, this study proposes an adaptive moment estimation-incorporated particle swarm optimization (Adam-PSO) algorithm that efficiently addresses the premature issues in a PSO algorithm. With achieved the hyper-parameter adaptation in an SGD-based LFA model, an Adam-PSO-based LFA (APL) model possessing hyper-parameter-free training is further implemented. Empirical studies on four HiDS matrices indicate that compared with state-of-the-art models with hyper-parameter adaptation settings, an APL model achieves the most efficient hyper-parameter-free training and highly competitive prediction accuracy for missing data of an HiDS matrix. Hence, it fits the need of real applications with high scalability and efficiency.

Latent factor analysis (LFA) is highly efficient in knowledge discovery from high-dimensional and sparse (HiDS) matrices frequently encountered in big data and web service related applications. A stochastic gradient descent (SGD) algorithm is commonly adopted as a learning algorithm for LFA owing to its high efficiency. However, its sequential nature makes it less scalable when processing large-scale data. Although an alternating SGD algorithm decouples an LFA process to achieve parallelization, its performance relies on its hyper-parameter selection that is highly expensive to tune. To address it, this paper presents three adaptive alternating SGD algorithms, thus leading to three Parallel Adaptive LFA (PAL) models for LFA on large-scale HiDS matrices. Experimental studies on HiDS matrices from industrial service applications show that the proposed PAL models perform significantly better than existing ones in terms of both convergence rate and computational efficiency, as well as achieve competitive prediction accuracy for missing data.

Latent factor (LF) model is an effective method for extracting useful knowledge from high-dimensional and sparse (HiDS) data generated by various industrial applications. Parallelized stochastic gradient descent (SGD) is widely used in building a parallelized LF model for handling large-scale HiDS data, but parallelized SGD suffers from slow convergence and considerable time cost. To address this issue, this study incorporates the principle of momentum into parallelized SGD, where momentum decay coefficient and learning rate are adjusted dynamically, and proposes a momentum-incorporated fast parallelized SGD (MFSGD) method to discover latent patterns from large-scale HiDS data. The experiments on two datasets show that the proposed MFSGD method outperforms state-of-the-art parallel SGD methods in terms of computational efficiency.

User-side Quality-of-Service (QoS) data are vital for efficient cloud service selection, while available QoS data in real applications are commonly described by a high-dimensional and Sparse (HiDS) matrix due to a) increasing users and services, and b) the impossibility of observing the full invoking mapping among users and services. A latent factor analysis (LFA) model has proven to be efficient in performing representation learning on such an HiDS QoS matrix. However, existing LFA models commonly adopt a first-order optimizer that cannot makes it approach the second-order stationary point of the learning objective, thereby resulting in accuracy loss. Aiming at addressing this issue, this study proposes to incorporate a Generalized Nesterov's Acceleration (GNA) method in to a Hessian-vector algorithm for LFA, thereby establishing a GNA-incorporated Hessian-vector-based LFA (GNHL) model with two-fold ideas: a) adopting the principle of a Hessian-vector method to acquire a proper Newton step efficiently, and b) incorporating a GNA method into its linear search for accelerating its convergence rate. Experimental results on six real QoS datasets demonstrate that a GNHL model outperforms state-of-the-art LFA models in generating highly accurate predictions for missing QoS data with low computational burden.

A recommender system (RS) relying on latent factor analysis usually adopts stochastic gradient descent (SGD) as its learning algorithm. However, owing to its serial mechanism, an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems. Aiming at addressing this issue, this study proposes a momentum-incorporated parallel stochastic gradient descent (MPSGD) algorithm, whose main idea is two-fold: a) implementing parallelization via a novel data-splitting strategy, and b) accelerating convergence rate by integrating momentum effects into its training process. With it, an MPSGD-based latent factor (MLF) model is achieved, which is capable of performing efficient and high-quality recommendations. Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm, an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

The multinomial logistic regression (MLR) model is widely used in statistics and machine learning. On the one hand, stochastic gradient descent (SGD) is the most common approach for determining the parameters of a such model in big data scenarios, due to its simplicity and low computational complexity property. Furthermore, SGD has proven convergence under reasonable conditions. However, SGD has slow sub-linear rates of convergence and it often reduces convergence speed due to the plateau phenomenon. On the other hand, stochastic natural gradient descent (SNGD), proposed by Amari, is a manifold optimization method shown to be Fisher efficient when it converges, but its convergence properties remain unproven and it is often computationally prohibitive for models with a large number of parameters. Here, we propose dual stochastic natural gradient descent (DSNGD), a stochastic optimization method for MLR based on manifold optimization concepts. In the discrete scenario, DSNGD (i) has linear per-iteration computational complexity in the number of parameters, and (ii) is proven to converge. To achieve (i) we leverage the dual flatness of the family of joint distributions for MLR to simplify computations. To ensure (ii) DSNGD builds on the foundational ideas of convergent stochastic natural gradient descent (CSNGD), a variant of SNGD with guaranteed convergence, using an independent sequence to construct a bounded approximation of the natural gradient. By generalizing a result from Sunehag et al., we prove that DSNGD converges in the discrete case and maintains linear computational complexity per iteration. Beyond its convergence property and linear computational complexity, DSNGD empirically demonstrates fast convergence comparable to SNGD, improves upon SGD performance, and exhibits stability where SNGD does not.

Non-negative latent factor (NLF) models well represent high-dimensional and sparse (HiDS) matrices filled with non-negative data, which are frequently encountered in industrial applications like recommender systems. However, current NLF models mostly adopt Euclidean distance in their objective function, which represents a special case of a β-divergence function. Hence, it is highly desired to design a β-divergence-based NLF ( β-NLF) model that uses a β-divergence function, and investigate its performance in recommender systems as β varies. To do so, we first model β-NLF's learning objective with a β-divergence function. Subsequently, we deduce a general single latent factor-dependent, non-negative and multiplicative update scheme for β-NLF, and then design an efficient β-NLF algorithm. The experimental results on HiDS matrices from industrial applications indicate that by carefully choosing the value of β, β-NLF outperforms an NLF model with Euclidean distance in terms of accuracy for missing data prediction without increasing computational time. The research outcomes show the necessity of using an optimal β-divergence function in order to achieve the best performance of an NLF model on HiDS matrices. Hence, the proposed model has both theoretical and application significance.

The latent factor analysis (LFA) model has been widely used to uncover latent relationships from high-dimensional sparse (HiDS) matrices. However, the performance of LFA depends largely on the hyper-parameter value used in the model training. Traditional hyper-parameter tuning methods such as grid search suffer from inefficiency and inaccuracy. In recent years, the particle swarm optimization (PSO) algorithm offers an intelligent approach to adaptively adjust the hyper-parameter of LFA. However, the global optimal solution of the hyper-parameter tuning problem is not fixed due to its dynamic decision space. Therefore, it is difficult for PSO to determine the best hyper-parameter for each training iteration. To address this problem, this paper proposes a novel hyper-parameter adaptive adjustment algorithm called dynamic stochastic reorientation PSO (DSR-PSO) that adapts to constantly changing decision spaces. By randomly adjusting the search directions of particles and perturbing the elite particles, the dynamic property of the DSR-PSO can be enhanced, so that the hyper-parameter can be adjusted in real time throughout the model training process. Furthermore, this paper proves the convergence of the DSR-PSO and gives its convergence condition by discussing the distribution of the characteristic roots. Finally, this paper proposes the DSR-PSO-based LFA (DPL) model by incorporating the DSR-PSO-based hyper-parameter adjustment into the LFA to promote its model training, and analyzes its complexity. Experimental results on benchmark datasets show that the proposed DPL surpasses state-of-the-art LFA models in terms of accuracy and efficiency.

Multivariate longitudinal data are used in a variety of research areas not only because they allow to analyze time trajectories of multiple indicators, but also to determine how these trajectories are influenced by other covariates. In this article, we propose a mixture of longitudinal factor analyzers. This model could be used to extract latent factors representing multiple longitudinal noisy indicators in heterogeneous longitudinal data and to study the impact of one or several covariates on these latent factors. One of the advantages of this model is that it allows for measurement non-invariance, which arises in practice when the factor structure varies between groups of individuals due to cultural or physiological differences. This is achieved by estimating different factor models for different latent classes. The proposed model could also be used to extract latent classes with different latent factor trajectories over time. Other advantages of the model include its ability to take into account heteroscedasticity of errors in the factor analysis model by estimating different error variances for different latent classes. We first define the mixture of longitudinal factor analyzers and its parameters. Then, we propose an EM algorithm to estimate these parameters. We propose a Bayesian information criterion to identify both the number of components in the mixture and the number of latent factors. We then discuss the comparability of the latent factors obtained between subjects in different latent groups. Finally, we apply the model to simulated and real data of patients with chronic postoperative pain.

We focus on sparse modelling of high-dimensional covariance matrices using Bayesian latent factor models. We propose a multiplicative gamma process shrinkage prior on the factor loadings which allows introduction of infinitely many factors, with the loadings increasingly shrunk towards zero as the column index increases. We use our prior on a parameter-expanded loading matrix to avoid the order dependence typical in factor analysis models and develop an efficient Gibbs sampler that scales well as data dimensionality increases. The gain in efficiency is achieved by the joint conjugacy property of the proposed prior, which allows block updating of the loadings matrix. We propose an adaptive Gibbs sampler for automatically truncating the infinite loading matrix through selection of the number of important factors. Theoretical results are provided on the support of the prior and truncation approximation bounds. A fast algorithm is proposed to produce approximate Bayes estimates. Latent factor regression methods are developed for prediction and variable selection in applications with high-dimensional correlated predictors. Operating characteristics are assessed through simulation studies, and the approach is applied to predict survival times from gene expression data.

This correlational study aims to identify the predictive latent factors of academic failure in a sample of university students not yet at risk of dropping out. 80 healthy students of the first academic year, voluntary recruited through the Ongoing Guidance service at “Roma Tre” University (female = 92%; mean age = 19 year, 8 months), took part in the study. They completed a battery of questionnaires using the online platform Limesurvey. We computed correlation coefficients (Pearson r) of the relationship between total drop-out score and the measure of cognitive and affective dimensions (STAI, STAXI, OQ-45.2, QLS-Italian QSA, ZTPI, AMS, SRKS-U, Questionnaire on the perceived quality of the interaction with the academic setting). Variables which survived Bonferroni correction for multiple comparisons were entered as predictors in a model of multiple regression analysis (stepwise method). The main results indicate that the Drop-Out score is significantly and positively correlated with State, Trait Anxiety and Anger-In scores, and also with the OQ-45.2 subscale (Symptom Distress, Interpersonal Relations and Social Role). Again, the Drop-Out score correlates positively with Anxiety and “Self-regulation” subscales of QLS (QSA). Finally, the students’ perception of integration within the University context (quality of social networking, utilization of students’ facilities and quality of the interaction with the teachers) appears correlated to specific attitudes or traits. In conclusion, there is evidence to consider the relevance of the social context as a predictive factor in developing the risk of dropping-out, considering the interaction of personality traits or attitudes as latent factors.

Mixture factor analysis is examined as a means of flexibly estimating nonnormally distributed continuous latent factors in the presence of both continuous and dichotomous observed variables. A simulation study compares mixture factor analysis with normal maximum likelihood (ML) latent factor modeling. Different results emerge for continuous versus dichotomous outcomes. For dichotomous outcomes, normal ML path estimates have bias that worsens as latent factor skew/kurtosis increases and does not diminish as sample size increases, whereas the mixture factor analysis model produces nearly unbiased estimators as sample sizes increase (500 and greater) and offers near nominal coverage probability. For continuous outcome variables, both methods produce factor loading estimates with minimal bias regardless of latent factor skew, but the mixture factor analysis is more efficient. The method is demonstrated using data motivated by a study on youth with cystic fibrosis examining predictors of treatment adherence. In summary, mixture factor analysis provides improvements over normal ML estimation in the presence of skewed/kurtotic latent factors, but due to variability in the estimator relating the latent factor to dichotomous outcomes and computational issues, the improvements were only fully realized, in this study, at larger sample sizes (500 and greater).

Executive function: Responses to aerobic exercise in Alzheimer's disease