Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization

Lee and Seung proposed nonnegative matrix factorization (NMF) algorithms to decompose patterns and images for structure retrieving. The NMF algorithms have been applied to various optimization problems. However, it is difficult to prove the convergence of this class of learning algorithms. This paper presents the global minima analysis of the NMF algorithms. In the analysis, invariant set is constructed so that the non-divergence of the algorithms can be guaranteed in the set. Using the features of linear equation systems and their solutions, the fixed points and convergence properties of the update algorithms are discussed in detail. The analysis shows that, although the cost function is not convex in both A and X together, it is possible to obtain the global minima from the particular learning algorithms. For different initializations, simulations are presented to confirm the analysis results.

Nonnegative matrix factorization (NMF) is a popular method for multivariate analysis of nonnegative data, the goal of which is decompose a data matrix into a product of two factor matrices with all entries in factor matrices restricted to be nonnegative. NMF was shown to be useful in a task of clustering (especially document clustering). In this paper we present an algorithm for orthogonal nonnegative matrix factorization, where an orthogonality constraint is imposed on the nonnegative decomposition of a term-document matrix. We develop multiplicative updates directly from true gradient on Stiefel manifold, whereas existing algorithms consider additive orthogonality constraints. Experiments on several different document data sets show our orthogonal NMF algorithms perform better in a task of clustering, compared to the standard NMF and an existing orthogonal NMF.KeywordsNonnegative Matrix FactorizationDocument ClusterOrthogonality ConstraintMultiplicative UpdateProbabilistic Latent Semantic IndexingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Face recognition is a challenging issue in field of multi-science, main contents of research is how to make computer have ability of face recognition face recognition technology involved in a lot, which is a key feature extraction and classification method, this paper focuses on study of related theory. Non-negative matrix factorization trapped MF) algorithm and local non-negative matrix factorization (LNMF) algorithm is a feature extraction method based on local features, has been successfully used in face recognition NMF algorithm in face recognition rate is low, although LNMF algorithm to a certain extent, improve recognition rate, but its price is to increase number of iterations. In addition, two algorithms have failed to solve good nonlinear separable problems the kernel method combined with LNMF algorithm, kernel local non-negative matrix factorization (KLNMF) algorithm, first by a nonlinear transformation of original space to high-dimensional space, making samples linearly separable, and then use LNMF algorithm to extract face features. In classification part, paper presents decision rules of classification of their own, and design based on NMF subspace classifier. DOI : http://dx.doi.org/10.11591/telkomnika.v12i3.4356 Full Text: PDF

Nonnegative matrix factorization (NMF) is a widely-used method for multivariate analysis of nonnegative data, the goal of which is decompose a data matrix into a basis matrix and an encoding variable matrix with all of these matrices allowed to have only nonnegative elements. In this paper we present simple algorithms for orthogonal NMF, where orthogonality constraints are imposed on basis matrix or encoding matrix. We develop multiplicative updates directly from the true gradient (natural gradient) in Stiefel manifold, whereas existing algorithms consider additive orthogonality constraints. Numerical experiments on face image data for a image representation task show that our orthogonal NMF algorithm preserves the orthogonality, while the goodness-of-fit (GOF) is minimized. We also apply our orthogonal NMF to a clustering task, showing that it works better than the original NMF, which is confirmed by experiments on several UCI repository data sets.

This article uses the projected gradient method (PG) for a non-negative matrix factorization problem (NMF), where one or both matrix factors must have orthonormal columns or rows. We penalize the orthonormality constraints and apply the PG method via a block coordinate descent approach. This means that at a certain time one matrix factor is fixed and the other is updated by moving along the steepest descent direction computed from the penalized objective function and projecting onto the space of non-negative matrices. Our method is tested on two sets of synthetic data for various values of penalty parameters. The performance is compared to the well-known multiplicative update (MU) method from Ding (2006), and with a modified global convergent variant of the MU algorithm recently proposed by Mirzal (2014). We provide extensive numerical results coupled with appropriate visualizations, which demonstrate that our method is very competitive and usually outperforms the other two methods.

Non-negative matrix factorization (NMF) is widely used in feature extraction and dimension reduction fields. Essentially, it is an optimization problem to determine two non-negative low rank matrices \(W_{m \times k}\) and \(H_{k \times n}\) for a given matrix \(A_{m \times n}\), satisfying \(A_{m \times n} \approx W_{m \times k}H_{k \times n}\). In this paper, a novel approach to improve the image decomposing and reconstruction effects by introducing the Singular Value Decomposing (SVD)-based initialization scheme of factor matrices W and H, and another measure called choosing rule to determine the optimum value of factor rank k, are proposed. The input image is first decomposed using SVD to get its singular values and corresponding eigenvectors. Then, the number of main components as the rank value k is extracted. Then, the singular values and corresponding eigenvectors are used to initialize W and H based on selected rank k. Finally, convergent results are obtained using multiplicative and additive update rules. However, iterative NMF algorithms’ convergence is very slow on most platforms limiting its practicality. To this end, a parallel implementation frame of described improved NMF algorithm using CUDA, a tool for algorithms parallelization on massively parallel processors, i.e., many-core graphics processors, is presented. Experimental results show that our approach can get better decomposing effect than traditional NMF implementations and dramatic accelerate rate comparing to serial schemes as well as existing distributed-system implementations.

Non-negative matrix factorization (NMF) approximates a given matrix as a product of two non-negative matrix factors. Multiplicative algorithms deliver reliable results, but they show slow convergence for high-dimensional data and may be stuck away from local minima. Gradient descent methods have better behavior, but only apply to smooth losses. For non-smooth losses such as the Kullback-Leibler (KL) loss, surprisingly, these methods are lacking. In this paper, we propose a first-order primal-dual algorithm for non-negative decomposition problems (one of the two factors is fixed) with the KL distance. All required computations may be obtained in closed form and we provide an efficient heuristic way to select step-sizes. By using alternating optimization, our algorithm readily extends to NMF and, on synthetic or real world data, it is either faster than existing algorithms, or leads to improved local optima, or both.

Hyperspectral unmixing is a powerful method of the remote sensing image mining that identifies the constituent materials and estimates the corresponding fractions from the mixture. We consider the application of nonnegative matrix factorization (NMF) for the mining and analysis of spectral data. In this paper, we develop two effective active set type NMF algorithms for hyperspectral unmixing. Because the factor matrices used in unmixing have sparse features, the active set strategy helps reduce the computational cost. These active set type algorithms for NMF is based on an alternating nonnegative constrained least squares (ANLS) and achieve a quadratic convergence rate under the reasonable assumptions. Finally, numerical tests demonstrate that these algorithms work well and that the function values decrease faster than those obtained with other algorithms.

ABSTRACTHyperspectral unmixing (HU) has drawn remarkable attention because it can decompose mixed pixels into a set of endmembers and abundance fractions. And the nonnegative matrix factorization (NMF) algorithm has been widely used for solving hyperspectral spectral unmixing problem. This letter proposes a Hessian graph regularized NMF (HGNMF) algorithm, which relies upon the construction of a Hessian graph representation, to solve the hyperspectral unmixing problem. According to the HGNMF algorithm, the smoothness in the estimated abundance maps is well promoted. Moreover, the optimized problem of HGNMF algorithm is also solved by employing the multiplicative updating rules. Compared to other algorithms, the proposed HGNMF algorithm demonstrates lots of advantages based on the simulated results of the synthetic data and real data sets.

The effects of fatigue on synergy of selected lower limb muscles during running

The significance of Non-negative Matrix Factorization ((NMF) in the field of automatic text summarization is rapidly increasing due to its interpretation and storage capabilities. Interpretation defines the ease at which the structure of high dimensional data can be understood. While, storage capability relates to the extent of data reduction process achieved by NMF. The parametric values that serve as input to the NMF process include initialization method, rank of factorization, sparseness measure and maximum iteration. These inputs are vital to the output produced by NMF. These parameters of NMF were not been considered in the existing literature on Automatic Text Summarization (ATS). This paper sheds light specifically on sparseness of NMF when applied to ATS and the impact it makes to the quality of the summary generated. In this study on NMF algorithm that supports sparseness is treated with various degree of sparsity and its performance impact on text summary generated is compared with other NMF algorithms without sparseness constraints.

Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods have become popular in modern data analysis problems due to their computational efficiency. Even though they have proved useful for many statistical models, the application of SG-MCMC to non-negative matrix factorization (NMF) models has not yet been extensively explored. In this study, we develop two parallel SG-MCMC algorithms for a broad range of NMF models. We exploit the conditional independence structure of the NMF models and utilize a stratified sub-sampling approach for enabling parallelization. We illustrate the proposed algorithms on an image restoration task and report encouraging results.

Smooth nonnegative matrix and tensor factorizations for robust multi-way data analysis

We present a numerical algorithm for nonnegative matrix factorization (NMF) problems under noisy separability. An NMF problem under separability can be stated as one of finding all vertices of the ...

For the quantitation of cerebral blood flow (CBF) using H/sub 2//sup 15/O PET, the measurement of arterial input function (AIF) is essential. In this study, for the simplified quantitation, we present a method for the blind and noninvasive extraction of carotid input function (CIF) from dynamic PET images. On 8 healthy volunteers, the PET scans of C/sup 15/O and H/sub 2//sup 15/O were sequentially performed with arterial blood sampling using detector. With the inhalation of C/sup 15/O gas, dynamic PET data was acquired. And then after the injection of H/sub 2//sup 15/O, dynamic PET scans of H/sub 2//sup 15/O was perform. For 4 subjects, PET data for both rest and Acetazolamide (ACZ)-induced stress states were acquired, respectively. In the transverse dynamic images of C/sup 15/O and H/sub 2//sup 15/O, the regions of dynamic images including carotid artery were selected by masking. Non-negative matrix factorization (NMF) algorithm was used to extract the CIF from the selected dynamic images. The partial volume correction estimated from dynamic C/sup 15/O image, was applied to the extracted CIF. Whole brain CBF was estimated by kinetic analysis based on single compartment model. The error was analyzed using the area under the curve (AUC) and estimated CBF. NMF provided the good separation of the component of CIF from others in both images of C/sup 15/O and H/sub 2//sup 15/O. The shape of estimated CIF by NMF was similar with AIF measured by blood sampling. The AUC between the measured AIF and the estimated CIF was not so different at both rest and ACZ states. The difference of CBF before and after the injection of ACZ was also same between the measured AIF and the estimated CIF. These results show that NMF technique may be a tool for the noninvasive extraction of carotid input function, and this carotid input function might be used in the noninvasive quantitation of CBF using H/sub 2//sup 15/O PET.