Randomized latent factor model for high-dimensional and sparse matrices from industrial applications

Non-negative latent factor (NLF) models well represent high-dimensional and sparse (HiDS) matrices filled with non-negative data, which are frequently encountered in industrial applications like recommender systems. However, current NLF models mostly adopt Euclidean distance in their objective function, which represents a special case of a β-divergence function. Hence, it is highly desired to design a β-divergence-based NLF ( β-NLF) model that uses a β-divergence function, and investigate its performance in recommender systems as β varies. To do so, we first model β-NLF's learning objective with a β-divergence function. Subsequently, we deduce a general single latent factor-dependent, non-negative and multiplicative update scheme for β-NLF, and then design an efficient β-NLF algorithm. The experimental results on HiDS matrices from industrial applications indicate that by carefully choosing the value of β, β-NLF outperforms an NLF model with Euclidean distance in terms of accuracy for missing data prediction without increasing computational time. The research outcomes show the necessity of using an optimal β-divergence function in order to achieve the best performance of an NLF model on HiDS matrices. Hence, the proposed model has both theoretical and application significance.

How to extract useful knowledge from a high-dimensional and sparse (HiDS) matrix efficiently is critical for many big data-related applications. A latent factor (LF) model has been widely adopted to address this problem. It commonly relies on an iterative learning algorithm like stochastic gradient descent. However, an algorithm of this kind commonly consumes many iterations to converge, resulting in considerable time cost on large-scale datasets. How to accelerate an LF model's training process without accuracy loss becomes a vital issue. To address it, this study innovatively proposes a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</u> ultilayered-and-randomized <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</u> atent <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</u> actor (MLF) model. Its main idea is two-fold: a) adopting randomized-learning to train LFs for implementing a ‘one-iteration’ training process for saving time; and 2) adopting the principle of a generally multilayered structure as in a deep forest or multilayered extreme learning machine to structure its LFs, thereby enhancing its representative learning ability. Empirical studies on six HiDS matrices from real applications demonstrate that compared with state-of-the-art LF models, an MLF model achieves significantly higher computational efficiency with satisfactory prediction accuracy. It has the potential to handle LF analysis on a large scale HiDS matrix with real-time requirements.

Latent factor (LF) model is an effective method for extracting useful knowledge from high-dimensional and sparse (HiDS) data generated by various industrial applications. Parallelized stochastic gradient descent (SGD) is widely used in building a parallelized LF model for handling large-scale HiDS data, but parallelized SGD suffers from slow convergence and considerable time cost. To address this issue, this study incorporates the principle of momentum into parallelized SGD, where momentum decay coefficient and learning rate are adjusted dynamically, and proposes a momentum-incorporated fast parallelized SGD (MFSGD) method to discover latent patterns from large-scale HiDS data. The experiments on two datasets show that the proposed MFSGD method outperforms state-of-the-art parallel SGD methods in terms of computational efficiency.

High-dimensional and sparse (HiDS) matrices are frequently adopted to describe the complex relationships in various big data-related systems and applications. A Position-transitional Latent Factor Analysis (PLFA) model can accurately and efficiently represent an HiDS matrix. However, its involved latent factors are optimized by particle-swarm-optimization-incorporated stochastic gradient descent with the specific gradient direction step-by-step, which may cause a suboptimal solution. To address this issue, this paper proposes a <u>S</u>equential-<u>G</u>roup-<u>D</u>ifferential-<u>E</u>volution (SGDE) algorithm to refine the latent factors optimized by a PLFA model, thereby achieving a highly-accurate SGDE-PLFA model to HiDS matrices. The main idea of SGDE-PLFA is two-fold. First, it divides the obtained latent factors by PLFA into row and column sub-groups to ensure the efficient low-dimensional optimization. Second, it initializes and optimizes these sub-group latent factors with our improved Differential Evolution algorithm to search more accurate solutions nearby the suboptimum. As demonstrated by the extensive experiments on four HiDS matrices, an SGDE-PLFA model outperforms the state-of-the-art models in representing HiDS matrices.

Joint latent factors and attributes to discover interpretable preferences in recommendation

Latent factor analysis (LFA) is highly efficient in knowledge discovery from high-dimensional and sparse (HiDS) matrices frequently encountered in big data and web service related applications. A stochastic gradient descent (SGD) algorithm is commonly adopted as a learning algorithm for LFA owing to its high efficiency. However, its sequential nature makes it less scalable when processing large-scale data. Although an alternating SGD algorithm decouples an LFA process to achieve parallelization, its performance relies on its hyper-parameter selection that is highly expensive to tune. To address it, this paper presents three adaptive alternating SGD algorithms, thus leading to three Parallel Adaptive LFA (PAL) models for LFA on large-scale HiDS matrices. Experimental studies on HiDS matrices from industrial service applications show that the proposed PAL models perform significantly better than existing ones in terms of both convergence rate and computational efficiency, as well as achieve competitive prediction accuracy for missing data.

Latent factor models have been widely used for recommendation. Most existing latent factor models mainly utilize the rating information between users and items, although some recently extended models add some auxiliary information to learn a unified latent factor between users and items. The unified latent factor only represents the latent features of users and items from the aspect of purchase history. However, the latent features of users and items may stem from different aspects, e.g., the brand-aspect and category-aspect of items. In this paper, we propose a Neural network based Aspect-level Collaborative Filtering model (NeuACF) to exploit different aspect latent factors. Through modelling rich objects and relations in recommender system as a heterogeneous information network, NeuACF first extracts different aspect-level similarity matrices of users and items through different meta-paths and then feeds an elaborately designed deep neural network with these matrices to learn aspect-level latent factors. Finally, the aspect-level latent factors are effectively fused with an attention mechanism for the top-N recommendation. Extensive experiments on three real datasets show that NeuACF significantly outperforms both existing latent factor models and recent neural network models.

Rating prediction using review texts with underlying sentiments

In this article, an improved double factorization-based symmetric and non-negative latent factor (Im-DF-SNLF) model is proposed to make the estimation for missing data in symmetric, high-dimensional, and sparse (SHiDS) matrices. The main idea of the Im-DF-SNLF model is fourfold: 1) considering the data variety in the practical engineering, non-negative latent factors (NLFs) in different cases are considered to better reflect the latent relationships between entries; 2) the l₂-norm regularization and the Lagrangian multiplier technique are simultaneously adopted to handle the overfitting and satisfy the non-negative constraint for latent factors (LFs); 3) the extragradient-based alternating direction (EGAD) method is utilized to accelerate the model training and rigidly guarantee the non-negativity of LFS; and 4) a rigorous proof is provided to show that, under the given assumption that the objective function is smooth and has a Lipschitz continuous gradient, the designed algorithm can find an ε-optimal solution within O(1/ε), and the upper bound of the learning rate is given by 1/2. Finally, experimental results on public datasets are given to demonstrate the effectiveness of our proposed Im-DF-SNLF model with EGAD.

This work introduces a novel estimation method, called LOVE, of the entries and structure of a loading matrix $A$ in a latent factor model $X=AZ+E$, for an observable random vector $X\\in \\mathbb{R}^{p}$, with correlated unobservable factors $Z\\in \\mathbb{R}^{K}$, with $K$ unknown, and uncorrelated noise $E$. Each row of $A$ is scaled, and allowed to be sparse. In order to identify the loading matrix $A$, we require the existence of pure variables, which are components of $X$ that are associated, via $A$, with one and only one latent factor. Despite the fact that the number of factors $K$, the number of the pure variables and their location are all unknown, we only require a mild condition on the covariance matrix of $Z$, and a minimum of only two pure variables per latent factor to show that $A$ is uniquely defined, up to signed permutations. Our proofs for model identifiability are constructive, and lead to our novel estimation method of the number of factors and of the set of pure variables, from a sample of size $n$ of observations on $X$. This is the first step of our LOVE algorithm, which is optimization-free, and has low computational complexity of order $p^{2}$. The second step of LOVE is an easily implementable linear program that estimates $A$. We prove that the resulting estimator is near minimax rate optimal for $A$, with respect to the $\\|\\ \\|_{\\infty ,q}$ loss, for $q\\geq 1$, up to logarithmic factors in $p$, and that it can be minimax-rate optimal in many cases of interest.\n¶ The model structure is motivated by the problem of overlapping variable clustering, ubiquitous in data science. We define the population level clusters as groups of those components of $X$ that are associated, via the matrix $A$, with the same unobservable latent factor, and multifactor association is allowed. Clusters are respectively anchored by the pure variables, and form overlapping subgroups of the $p$-dimensional random vector $X$. The Latent model approach to OVErlapping clustering is reflected in the name of our algorithm, LOVE.\n¶ The third step of LOVE estimates the clusters from the support of the columns of the estimated $A$. We guarantee cluster recovery with zero false positive proportion, and with false negative proportion control. The practical relevance of LOVE is illustrated through the analysis of a RNA-seq data set, devoted to determining the functional annotation of genes with unknown function.

Mixture factor analysis is examined as a means of flexibly estimating nonnormally distributed continuous latent factors in the presence of both continuous and dichotomous observed variables. A simulation study compares mixture factor analysis with normal maximum likelihood (ML) latent factor modeling. Different results emerge for continuous versus dichotomous outcomes. For dichotomous outcomes, normal ML path estimates have bias that worsens as latent factor skew/kurtosis increases and does not diminish as sample size increases, whereas the mixture factor analysis model produces nearly unbiased estimators as sample sizes increase (500 and greater) and offers near nominal coverage probability. For continuous outcome variables, both methods produce factor loading estimates with minimal bias regardless of latent factor skew, but the mixture factor analysis is more efficient. The method is demonstrated using data motivated by a study on youth with cystic fibrosis examining predictors of treatment adherence. In summary, mixture factor analysis provides improvements over normal ML estimation in the presence of skewed/kurtotic latent factors, but due to variability in the estimator relating the latent factor to dichotomous outcomes and computational issues, the improvements were only fully realized, in this study, at larger sample sizes (500 and greater).

Latent factor models are valuable in bioinformatics for accounting for unmeasured variation alongside observed covariates. Yet many methods struggle to separate known effects from latent structure and to handle losses beyond standard regression. We present a unified framework that augments row and column predictors with a low-rank latent component, jointly modeling measured effects and residual variation. To remove ambiguity in estimating observed and latent effects, we impose a carefully designed set of orthogonality constraints on the coefficient and latent factor matrices, relative to the spans of the predictor matrices. These constraints ensure identifiability, yield a decomposition in which the latent term captures only variation unexplained by the covariates, and improve interpretability. An efficient algorithm handles general non-quadratic losses via surrogates with monotone descent. Each iteration updates the latent term by truncated singular value decomposition of a doubly projected residual and refines coefficients by projections. The number of latent factors is selected by applying an elbow rule to a degrees-of-freedom-adjusted information criterion. A parametric bootstrap provides valid inference on feature-outcome associations under the regularized low-rank structure. Applied to real pharmacogenomic data, the method recovers biologically coherent gene-drug associations missed by standard factor models, such as the EGFR-inhibitor link, highlights novel candidates with plausible mechanisms, and reveals gene programs aligned with compound modes of action, including a latent unfolded-protein-response module affecting drug sensitivity. These results support the framework’s utility for precision oncology, yielding stronger biomarkers for patient stratification and deeper insight into drug resistance mechanisms.

One important problem of recommender system is rating prediction. In this paper, we use the movie rating data from MovieLens as an example to show how to use users’ attributes to improve the accuracy of rating prediction. Through data analysis, we observe that users having similar attributes tend to share more similar preferences and users with a special attribute have their own preferred items. Based on the two observations, we assume that a user’s rating to an item is determined by both the user intrinsic characteristics and the user common characteristics. Using the widely adopted latent factor model for rating prediction, in our proposed solution, we use two kinds of latent factors to model a user: one for the user intrinsic characteristics and the other for the user common characteristics. The latter encodes the influence of users’ attributes which include user age, gender and occupation. On the other hand, we jointly use user attributes or item category information and rating data for calculating similarity of users or items. The similarity calculating results are used in our proposed latent factor model as a regularization term to regularize users or items latent factors gap. Experimental results on MovieLens show that by incorporating users’ attributes influences, much lower prediction error is achieved than the state-of-the-art models. The prediction error is further reduced by incorporating influences from item category popularity and item popularity.

A recommender system (RS) relying on latent factor analysis usually adopts stochastic gradient descent (SGD) as its learning algorithm. However, owing to its serial mechanism, an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems. Aiming at addressing this issue, this study proposes a momentum-incorporated parallel stochastic gradient descent (MPSGD) algorithm, whose main idea is two-fold: a) implementing parallelization via a novel data-splitting strategy, and b) accelerating convergence rate by integrating momentum effects into its training process. With it, an MPSGD-based latent factor (MLF) model is achieved, which is capable of performing efficient and high-quality recommendations. Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm, an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

With the rapid development of basic computing and storage facilities, the big-data-related industrial applications, i.e., recommendation system [1–5], cloud services [6–10], social networks [11–16], and wireless sensor networks [17–22], continue to expand. Consequently, the data involved in these applications grow exponentially. Such data commonly contains a mass of entities. With the increasing number of involved entities, it becomes impossible to observe their whole interaction mapping. Hence, a High-dimensional and sparse (HiDS) matrix [23–31, 60] is commonly adopted to describe such specific relationships among entities. For instance, the Douban matrix [32] collected by the Chinese largest online book, movie and music database includes 129,490 users and 58,541 items. However, it only contains 16,830,839 known ratings and the density is 0.22%.