Highly scalable parallel algorithms for sparse matrix factorization

Non-negative Matrix Factorization (NMF) is widely used in image clustering; however, it has inherent limitations, including its unsupervised nature, lack of sparsity constraints, inability to leverage label information, and difficulty in capturing the geometric structure and nonlinear characteristics of data. To address these limitations, this paper proposes a Sparse Kernel Graph-regularized Discriminant Non-negative Matrix Factorization (SKGDNMF) algorithm. The algorithm innovatively adopts a dual normalization strategy, which involves column normalization for the basis matrix and row normalization for the coefficient matrix, and integrates spectral clustering to construct an end-to-end deep clustering framework. By applying sparse regularization to the coefficient matrix, the model’s robustness is significantly improved; this regularization forms complementary optimization with graph regularization, thereby effectively alleviating overfitting. A three-dimensional golden-section parameter optimization method is employed to determine key parameters, which enhances the algorithm’s practicality. Comparative experiments conducted on multiple datasets show that SKGDNMF significantly outperforms mainstream algorithms in terms of sparsity, robustness, and clustering performance, indicating its superior effectiveness for image clustering tasks.

The explosive growth of data has caused users to spend considerable time and effort finding the items they need. Various recommender systems have been created to provide convenience for users. This paper proposes a low-rank and sparse matrix factorization with prior relations (LSMF-PR) recommendation model, which predicts users’ ratings for items through a sum of the learned low-rank matrix and sparse matrix. Thus, unlike traditional matrix factorization approaches, our method can alleviate the error propagation produced by intermediate outputs. The LSMF-PR integrates user relationships and item relationships as prior information. User relationships in different recommendation scenarios are extracted by the corresponding social relations of the users, and item relationships are obtained from the similarity of the item content. Therefore, the sparsity and cold start problems can be effectively reduced with prior information. Furthermore, our model has better interpretability since it reveals the low-rank and sparse features of the ratings. Experiments are conducted on four real-world datasets to validate the performance of our proposed method.

This paper proposes parallel method of non-negative large sparse matrix factorization - very popular technique in computational linguistics. Proposed blocks factorization approach using Latent Dirichlet allocation, for speedup of non-negative sparse extra-large linguistic matrix and tensor factorization. The proposed method also supports the scalability of the model.

In many scientific computing applications, sparse Cholesky factorization is used to solve large sparse linear equations in distributed environment. GPU computing is a new way to solve the problem. However, sparse Cholesky factorization on GPU is hardly to achieve excellent performance due to the structure irregularity of matrix and the low GPU resource utilization. A hybrid CPU-GPU implementation of sparse Cholesky factorization is proposed based on multifrontal method. A large sparse coefficient matrix is decomposed into a series of small dense matrices (frontal matrices) in the method, and then multiple GEMM (General Matrix-matrix Multiplication) operations are computed on them. GEMMs are the main operations in sparse Cholesky factorization, but they are hardly to perform better in parallel on GPU. In order to improve the performance, the scheme of multiple task queues is adopted to perform multiple GEMMs parallelized with multifrontal method; all GEMM tasks are scheduled dynamically on GPU and CPU based on computation scales for load balance and computing-time reduction. Experimental results show that the approach can outperform the implementations of cuBLAS, achieving up to 1.98× speedup on GTX460 (Fermi micro-architecture) and 3.06× speedup on K20m (Kepler micro-architecture), respectively.

In this paper, the generation of sequency-ordered complex Hadamard transform (SCHT) based on the complex Rademacher matrices is presented. The exponential form of SCHT is also derived, and the proof for the unitary property of SCHT is given. Using the sparse matrix factorization, the fast and efficient algorithm to compute the SCHT transform is developed, and its computation load is described. Certain properties of the SCHT matrices are derived and analyzed with the discussion of SCHT applications in spectrum analysis and image watermarking. Relations of SCHT with fast Fourier transform (FFT) and unified complex Hadamard transform (UCHT) are discussed.

Zero-shot learning, a special case of unsupervised domain adaptation where the source and target domains have disjoint label spaces, has become increasingly popular in the computer vision community. In this paper, we propose a novel zero-shot learning method based on discriminative sparse non-negative matrix factorization. The proposed approach aims to identify a set of common high-level semantic components across the two domains via non-negative sparse matrix factorization, while enforcing the representation vectors of the images in this common component-based space to be discriminatively aligned with the attribute-based label representation vectors. To fully exploit the aligned semantic information contained in the learned representation vectors of the instances, we develop a label propagation based testing procedure to classify the unlabeled instances from the unseen classes in the target domain. We conduct experiments on four standard zero-shot learning image datasets, by comparing the proposed approach to the state-of-the-art zero-shot learning methods. The empirical results demonstrate the efficacy of the proposed approach.

We study the problem of large-scale network embedding, which aims to learn\nlatent representations for network mining applications. Previous research shows\nthat 1) popular network embedding benchmarks, such as DeepWalk, are in essence\nimplicitly factorizing a matrix with a closed form, and 2)the explicit\nfactorization of such matrix generates more powerful embeddings than existing\nmethods. However, directly constructing and factorizing this matrix---which is\ndense---is prohibitively expensive in terms of both time and space, making it\nnot scalable for large networks.\n In this work, we present the algorithm of large-scale network embedding as\nsparse matrix factorization (NetSMF). NetSMF leverages theories from spectral\nsparsification to efficiently sparsify the aforementioned dense matrix,\nenabling significantly improved efficiency in embedding learning. The\nsparsified matrix is spectrally close to the original dense one with a\ntheoretically bounded approximation error, which helps maintain the\nrepresentation power of the learned embeddings. We conduct experiments on\nnetworks of various scales and types. Results show that among both popular\nbenchmarks and factorization based methods, NetSMF is the only method that\nachieves both high efficiency and effectiveness. We show that NetSMF requires\nonly 24 hours to generate effective embeddings for a large-scale academic\ncollaboration network with tens of millions of nodes, while it would cost\nDeepWalk months and is computationally infeasible for the dense matrix\nfactorization solution. The source code of NetSMF is publicly available\n(https://github.com/xptree/NetSMF).\n

Solving large sparse linear systems is often the most computationally intensive component of many scientific computing applications. In the past, sparse multifrontal direct factorization has been shown to scale to thousands of processors on dedicated supercomputers resulting in a substantial reduction in computational time. In recent years, an alternative computing paradigm based on GPUs has gained prominence, primarily due to its affordability, power-efficiency, and the potential to achieve significant speedup relative to desktop performance on regular and structured parallel applications. However, sparse matrix factorization on GPUs has not been explored sufficiently due to the complexity involved in an efficient implementation and concerns of low GPU utilization. In this paper, we present an adaptive hybrid approach for accelerating sparse multifrontal factorization based on a judicious exploitation of the processing power of the host CPU and GPU. We present four different policies for distributing and scheduling the workload between the host CPU and the GPU, and propose a mechanism for a runtime selection of the appropriate policy for each step of sparse Cholesky factorization. This mechanism relies on auto-tuning based on modeling the best policy predictor as a parametric classifier. We estimate the classifier parameters from the available empirical computation time data such that the expected computation time is minimized. This approach is readily adaptable for using the current or an extended set of policies for different CPU-GPU combinations as well as for different combinations of dense kernels for both the CPU and the GPU.

An important problem of matrix completion/approximation based on Matrix Factorization (MF) algorithms is the existence of multiple global optima; this problem is especially serious when the matrix is sparse, which is common in real-world applications such as personalized recommender systems. In this work, we clarify data sparsity by bounding the solution space of MF algorithms. We present the conditions that an MF algorithm should satisfy for reliable completion of the unobservables, and we further propose to augment current MF algorithms with extra constraints constructed by compressive sampling on the unobserved values, which is well-motivated by the theoretical analysis. Model learning and optimal solution searching is conducted in a properly reduced solution space to achieve more accurate and efficient rating prediction performances. We implemented the proposed algorithms in the Map-Reduce framework, and comprehensive experimental results on Yelp and Dianping datasets verified the effectiveness and efficiency of the augmented matrix factorization algorithms.

As with many other linear algebra algorithms, devising a portable implementation of sparse Cholesky factorization that performs well on the broad range of computer architectures currently available is a formidable challenge. Even after limiting our attention to machines with only one processor, as we have done in this report, there are still several interesting issues to consider. For dense matrices, it is well known that block factorization algorithms are the best means of achieving this goal. We take this approach for sparse factorization as well. This paper has two primary goals. First, we examine two sparse Cholesky factorization algorithms, the multifrontal method and a blocked left-looking sparse Cholesky method, in a systematic and consistent fashion, both to illustrate the strengths of the blocking techniques in general and to obtain a fair evaluation of the two approaches. Second, we assess the impact of various implementation techniques on time and storage efficiency, paying particularly close attention to the work-storage requirement of the two methods and their variants.

With the rapid growth of scientific publications, it is hard for researchers to acquire appropriate papers that meet their expectations. Recommendation system for scientific articles is an essential technology to overcome this problem. In this paper, we propose a novel low-rank and sparse matrix factorization-based paper recommendation (LSMFPRec) method for authors. The proposed method seamlessly combines low-rank and sparse matrix factorization method with fine-grained paper and author affinity matrixes that are extracted from heterogeneous scientific network. Thus, it can effectively alleviate the sparsity and cold start problems that exist in traditional matrix factorization based collaborative filtering methods. Moreover, LSMFPRec can significantly reduce the error propagated from intermediate outputs. In addition, the proposed method essentially captures the low-rank and sparse characteristics that exist in scientific rating activities; therefore, it can generate more reasonable predicted ratings for influential and uninfluential papers. The effectiveness of the proposed LSMFPRec is demonstrated by the recommendation evaluation conducted on the AAN and CiteULike data sets.

There is often a trade-off between preserving sparsity and numerical stability in sparse matrix factorizations. In applications like the direct solution of Equality Constrained Least Squares problem, the accurate detection of the rank of a large and sparse constraint matrix is a key issue. Column pivoting is not suitable for distributed memory machines because it forces the program into a lock-step mode, preventing any overlapping of computations. So factorization algorithms on such machines need to use a reliable, yet inexpensive incremental condition estimator to decide on which columns to be included. We describe an incremental condition estimator that can be used during a sparse QR factorization. We show that it is quite reliable and is well suited for use on parallel machines. We supply experimental results to support its effectiveness as well as suitability for parallel architectures.

In this article, the fast algorithms for the discrete Tchebichef transform (DTT) are proposed for input sequences of lengths in the range from 3 to 8. At present, DTT is widely applied in signal processing, image compression, and video coding. The review of the articles related to fast DTT algorithms has shown that such algorithms are mainly developed for input signal lengths 4 and 8. However, several problems exist for which signal and image processing with different apertures is required. To avoid this shortcoming, the structural approach and a sparse matrix factorization are applied in this paper to develop fast real DTT algorithms for short-length input signals. According to the structural approach, the rows and columns of the transform matrix are rearranged, possibly by changing the signs of some rows or columns. Next, the matched submatrix templates are extracted from the matrix structure and decomposed into a matrix product to construct the factorization of an initial matrix. A sparse matrix factorization assumes that the butterfly architecture can be extracted from the transform matrix. Combining the structural approach with a sparse matrix factorization, we obtained the matrix representation with reduced computational complexity. Based on the obtained matrix representation, the fast algorithms were developed for the real DTT via the data flow graphs. The fast algorithms for integer DTT can be easily obtained using the constructed data flow graphs. To confirm the correctness of the designed algorithms, the MATLAB R2023b software was applied. The constructed factorizations of the real DTT matrices reduce the number of multiplication operations by 78% on average compared to the direct matrix-vector product at signal lengths in the range from 3 to 8. The number of additions decreased by 5% on average within the same signal length range.

Sparse matrix factorization robust to sample sharing across GWASs reveals interpretable genetic components

Block-oriented sparse Cholesky factorization decomposes a sparse matrix into rectangular subblocks; each block can then be handled as a computational unit in order to increase data reuse in a hierarchical memory system. Also, the factorization method increases the degree of concurrency and reduces the overall communication volume so that it performs more efficiently on a distributed-memory multiprocessor system than the customary column-oriented factorization method. But until now, mapping of blocks to processors has been designed for load balance with restricted communication patterns. In this paper, we represent tasks using a block dependency DAG that represents the execution behavior of block sparse Cholesky factorization in a distributed-memory system. Since the characteristics of tasks for block Cholesky factorization are different from those of the conventional parallel task model, we propose a new task scheduling algorithm using a block dependency DAG. The proposed algorithm consists of two stages: early-start clustering, and affined cluster mapping (ACM). The early-start clustering stage is used to cluster tasks while preserving the earliest start time of a task without limiting parallelism. After task clustering, the ACM stage allocates clusters to processors considering both communication cost and load balance. Experimental results on