Network Embedding as Matrix Factorization

Network embedding approaches have recently attracted considerable interest as\nthey learn low-dimensional vector representations of nodes. Embeddings based on\nthe matrix factorization are effective but they are usually computationally\nexpensive due to the eigen-decomposition step. In this paper, we propose a\nRandom RangE FInder based Network Embedding (REFINE) algorithm, which can\nperform embedding on one million of nodes (YouTube) within 30 seconds in a\nsingle thread. REFINE is 10x faster than ProNE, which is 10-400x faster than\nother methods such as LINE, DeepWalk, Node2Vec, GraRep, and Hope. Firstly, we\nformulate our network embedding approach as a skip-gram model, but with an\northogonal constraint, and we reformulate it into the matrix factorization\nproblem. Instead of using randomized tSVD (truncated SVD) as other methods, we\nemploy the Randomized Blocked QR decomposition to obtain the node\nrepresentation fast. Moreover, we design a simple but efficient spectral filter\nfor network enhancement to obtain higher-order information for node\nrepresentation. Experimental results prove that REFINE is very efficient on\ndatasets of different sizes (from thousand to million of nodes/edges) for node\nclassification, while enjoying a good performance.\n

Combining Matrix Factorization (MF) with Network Embedding (NE) has been a promising solution to social recommender systems. However, in most of the current combined schemes, the user-specific linking proportions learned by NE are fed to the downstream MF, but not reverse, which is sub-optimal as the rating information is not utilized to discover the linking features for users. Furthermore, the existing combined models mainly focus on enhancing the representation learning for users by exploiting user–user network, yet ignore the representation improvement for items. In this paper, we propose a novel social recommendation scheme, called MF with dual-network collaborative embedding (MF-decoding), which jointly optimizes an integrated objective function of MF and NE, in which both MF and NE tasks can be mutually reinforced in a unified learning process. In particular, the explicit user–user network and the implicit item–item network are collaboratively used by MF-decoding to enhance the representation learning for users and items simultaneously. Our encouraging experimental results on three benchmarks validate the superiority of the proposed MF-decoding model over state-of-the-art social recommendation methods.

Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skip-gram model without a proper negative sampler. By performing an insightful analysis from the stochastic gradient descent (SGD) learning perspective, we demonstrate, both theoretically and intuitively, negative samples with larger inner product scores are more informative than those with lower scores for the SGD learner in terms of both convergence rate and accuracy. Understanding this, we propose an alternative sampling algorithm that dynamically selects informative negative samples during each SGD update. More importantly, the proposed sampler accounts for multi-dimensional self-embedded features during the sampling process, which essentially makes it more effective than the original popularity-based (one-dimensional) sampler. Empirical experiments further verify our observations and show that our fine-grained samplers gain significant improvement over the existing ones without increasing computational complexity.

In this paper we extend the current state-of-the-art matrix factorization method for recommendations to general probability distributions. As shown in previous work, the standard method called Probabilistic Matrix Factorization is based on a normal distribution assumption. While there exists work in which this method is extended to other distributions, these extensions are restrictive and we experimentally show on the basis of a real data set that it is worthwhile considering more general distributions which have not been used in the literature. Our contribution lies in providing a flexible and easy-to-use framework for matrix factorization with almost no limitation on the form of the distribution used. Our approach is based on maximum likelihood estimation and a key ingredient of our proposed method is automatic differentiation. This allows for the automatic derivation of the corresponding optimization algorithm, without the need to derive it manually for each distributional assumption while simultaneously being computationally efficient. Thus, with our method it is very easy to use a wide range of even complicated distributions for any data set.

Matrix Factorization (MF) is one of the most successful Collaborative Filtering (CF) techniques used in recommender systems due to its effectiveness and ability to deal with very large user-item rating matrix. However, when the rating matrix sparseness increases its performance deteriorates. Expanding MF to include side-information of users and items has been shown by many researchers both to improve general recommendation performance and to help alleviate the data-sparsity and cold-start issues in CF. In regard to item feature side-information, most schemes incorporate this information through a two stage process: intermediate results (e.g., on item similarity) are first computed based on item attributes; these are then combined with MF. In this paper, focussing on item side-information, we propose a model that directly incorporates item features into the MF framework in a single step process. The model, which we name FeatureMF, does this by projecting every available attribute datum in each of the item features into the same latent factor space with users and items, thereby in effect enriching item representation in MF. Results are presented of comparative performance experiments of the model against three state-of-the-art item information enriched models, as well as against four reference benchmark models, using two public real-world datasets, Douban and Yelp, with four training:test ratio scenarios each. It is shown to yield the best recommendation performance over all these models across all contexts including data-sparsity situations, in particular, achieving over 0.9% to over 6.5% MAE recommendation performance improvement over the next best model, HERec. FeatureMF is also found to alleviate cold start and to scale well, almost linearly, in regard to computational time, as a function of dataset size.

OLMNE+FT: Multiplex network embedding based on overlapping links

Representation learning via Dual-Autoencoder for recommendation

Network embedding which aims to embed a given network into a low-dimensional vector space has been proved effective in various network analysis and mining tasks such as node classification, link prediction and network visualization. The emerging network embedding methods have shifted of emphasis in utilizing mature deep learning models. The neural-network based network embedding has become a mainstream solution because of its high efficiency and capability of preserving the nonlinear characteristics of the network. In this paper, we propose Adversarial Network Embedding using Structural Similarity (ANESS), a novel, versatile, low-complexity GAN-based network embedding model which utilizes the inherent vertex-to-vertex structural similarity attribute of the network. ANESS learns robustness and effective vertex embeddings via a adversarial training procedure. Specifically, our method aims to exploit the strengths of generative adversarial networks in generating high-quality samples and utilize the structural similarity identity of vertexes to learn the latent representations of a network. Meanwhile, ANESS can dynamically update the strategy of generating samples during each training iteration. The extensive experiments have been conducted on the several benchmark network datasets, and empirical results demonstrate that ANESS significantly outperforms other state-of-the-art network embedding methods.

We study the problem of personalized Quality of Service (QoS) estimation for web services. State-of-the-art methods use matrix factorization or collaborative prediction to estimate web service response times and throughput for each user based on partial measurements collected from past invocations. We point out that in reality, both the response times and through-put of web services follow highly skewed distributions. In this case, the conditional mean QoS estimates generated by traditional matrix completion approaches can be heavily biased toward a few outliers, leading to poor web service recommendation performance. In this paper, we propose the Quantile Matrix Factorization (QMF) technique for web service recommendation by introducing quantile regression into the matrix factorization framework. We propose a novel and efficient algorithm based on Iterative Reweighted Least Squares (IRLS) to solve the QMF problem involving a non-smooth objective function. We further extend the proposed QMF approach to take into account user and service side attributes. Extensive evaluation based on a large-scale QoS dataset has shown that our schemes significantly outperform various state-of-the-art web service QoS estimation schemes in terms of personalized recommendation performance.

Saliency detection is used to identify the most important and informative area in a scene, and it is widely used in various vision tasks, including image quality assessment, image matching, and object recognition. Manifold ranking (MR) has been used to great effect for the saliency detection, since it not only incorporates the local spatial information but also utilizes the labeling information from background queries. However, MR completely ignores the feature information extracted from each superpixel. In this paper, we propose an MR-based matrix factorization (MRMF) method to overcome this limitation. MRMF models the ranking problem in the matrix factorization framework and embeds query sample labels in the coefficients. By incorporating spatial information and embedding labels, MRMF enforces similar saliency values on neighboring superpixels and ranks superpixels according to the learned coefficients. We prove that the MRMF has good generalizability, and develops an efficient optimization algorithm based on the Nesterov method. Experiments using popular benchmark data sets illustrate the promise of MRMF compared with the other state-of-the-art saliency detection methods.

Foreground detection is one of the key steps in computer vision applications. Many foreground and background models have been proposed and achieved promising performance in static scenes. However, due to challenges such as dynamic background, irregular movement, and noise, most algorithms degrade sharply in complex scenes. To address the problem, we propose a motion-aware structured matrix factorization approach (MSMF), which integrates the structural and spatiotemporal motion information into a unified sparse-low-rank matrix factorization framework. Technologically, it has three main contributions: First, a variant of structured sparsity-inducing norm is proposed to constrain both structure and sparsity of foreground. The model is robust to the statistical variability of the underlying foreground pixels in complex scenes. Second, to capture the ambiguous pixels, a spatiotemporal cube-based motion trajectory is extracted for assisting matrix factorization. Finally, to solve the optimization problem of structured matrix factorization, we develop an augmented Lagrange multiplier method with the alternating direction strategy and Douglas-Rachford monotone operator splitting algorithm. Experiments demonstrate that the proposed approach achieves impressive performance in separating irregular moving foreground while suppressing the dynamic background and the noise, and outperforms some state-of-the-art algorithms.

We study the problem of large-scale network embedding, which aims to learn\nlatent representations for network mining applications. Previous research shows\nthat 1) popular network embedding benchmarks, such as DeepWalk, are in essence\nimplicitly factorizing a matrix with a closed form, and 2)the explicit\nfactorization of such matrix generates more powerful embeddings than existing\nmethods. However, directly constructing and factorizing this matrix---which is\ndense---is prohibitively expensive in terms of both time and space, making it\nnot scalable for large networks.\n In this work, we present the algorithm of large-scale network embedding as\nsparse matrix factorization (NetSMF). NetSMF leverages theories from spectral\nsparsification to efficiently sparsify the aforementioned dense matrix,\nenabling significantly improved efficiency in embedding learning. The\nsparsified matrix is spectrally close to the original dense one with a\ntheoretically bounded approximation error, which helps maintain the\nrepresentation power of the learned embeddings. We conduct experiments on\nnetworks of various scales and types. Results show that among both popular\nbenchmarks and factorization based methods, NetSMF is the only method that\nachieves both high efficiency and effectiveness. We show that NetSMF requires\nonly 24 hours to generate effective embeddings for a large-scale academic\ncollaboration network with tens of millions of nodes, while it would cost\nDeepWalk months and is computationally infeasible for the dense matrix\nfactorization solution. The source code of NetSMF is publicly available\n(https://github.com/xptree/NetSMF).\n

Although multi-view learning has made significant progress over the past few decades, it is still challenging due to the difficulty in modeling complex correlations among different views, especially under the context of view missing. To address the challenge, we propose a novel framework termed Cross Partial Multi-View Networks (CPM-Nets), which aims to fully and flexibly take advantage of multiple partial views. We first provide a formal definition of completeness and versatility for multi-view representation and then theoretically prove the versatility of the learned latent representations. For completeness, the task of learning latent multi-view representation is specifically translated to a degradation process by mimicking data transmission, such that the optimal tradeoff between consistency and complementarity across different views can be implicitly achieved. Equipped with adversarial strategy, our model stably imputes missing views, encoding information from all views for each sample to be encoded into latent representation to further enhance the completeness. Furthermore, a nonparametric classification loss is introduced to produce structured representations and prevent overfitting, which endows the algorithm with promising generalization under view-missing cases. Extensive experimental results validate the effectiveness of our algorithm over existing state of the arts for classification, representation learning and data imputation.

Network embedding, aiming to learn the low-dimensional representations of nodes in networks, is of paramount importance in many real applications. One basic requirement of network embedding is to preserve the structure and inherent properties of the networks. While previous network embedding methods primarily preserve the microscopic structure, such as the first- and second-order proximities of nodes, the mesoscopic community structure, which is one of the most prominent feature of networks, is largely ignored. In this paper, we propose a novel Modularized Nonnegative Matrix Factorization (M-NMF) model to incorporate the community structure into network embedding. We exploit the consensus relationship between the representations of nodes and community structure, and then jointly optimize NMF based representation learning model and modularity based community detection model in a unified framework, which enables the learned representations of nodes to preserve both of the microscopic and community structures. We also provide efficient updating rules to infer the parameters of our model, together with the correctness and convergence guarantees. Extensive experimental results on a variety of real-world networks show the superior performance of the proposed method over the state-of-the-arts.

Recently, network embedding has received a large amount of attention in network analysis. Although some network embedding methods have been developed from different perspectives, on one hand, most of the existing methods only focus on leveraging the plain network structure, ignoring the abundant attribute information of nodes. On the other hand, for some methods integrating the attribute information, only the lower-order proximities (e.g., microscopic proximity structure) are taken into account, which may suffer if there exists the sparsity issue and the attribute information is noisy. To overcome this problem, the attribute information and mesoscopic community structure are utilized. In this article, we propose a novel network embedding method termed Attributed Network Embedding with Micro-Meso structure, which is capable of preserving both the attribute information and the structural information including the microscopic proximity structure and mesoscopic community structure. In particular, both the microscopic proximity structure and node attributes are factorized by Nonnegative Matrix Factorization (NMF), from which the low-dimensional node representations can be obtained. For the mesoscopic community structure, a community membership strength matrix is inferred by a generative model (i.e., BigCLAM) or modularity from the linkage structure, which is then factorized by NMF to obtain the low-dimensional node representations. The three components are jointly correlated by the low-dimensional node representations, from which two objective functions (i.e., ANEM_B and ANEM_M) can be defined. Two efficient alternating optimization schemes are proposed to solve the optimization problems. Extensive experiments have been conducted to confirm the superior performance of the proposed models over the state-of-the-art network embedding methods.