Parallelized Stochastic Gradient Markov Chain Monte Carlo algorithms for non-negative matrix factorization

Deep network embedding with dimension selection

We consider the problem of inferring an unknown number of clusters in multinomial count data, by estimating finite mixtures of multinomial distributions with or without covariates. Both Maximum Likelihood (ML) as well as Bayesian estimation are taken into account. Under a Maximum Likelihood approach, we provide an Expectation–Maximization (EM) algorithm which exploits a careful initialization procedure combined with a ridge-stabilized implementation of the Newton-Raphson method in the M-step. Under a Bayesian setup, a stochastic gradient Markov chain Monte Carlo (MCMC) algorithm embedded within a prior parallel tempering scheme is devised. The number of clusters is selected according to the Integrated Completed Likelihood criterion in the ML approach and estimating the number of non-empty components in overfitting mixture models in the Bayesian case. Our method is illustrated in simulated data and applied to two real datasets. The proposed methods are implemented in a contributed package, available online.

Deep learning models are traditionally used in big data scenarios. When there is not enough training data to fit a large model, transfer learning re-purpose the learned features from an existing model and re-train the lower layers for the new task. Bayesian inference techniques can be used to capture the uncertainty of the new model but it comes with a high computational cost. In this paper, the run time performance of an Stochastic Gradient Markov Chain Monte Carlo method using two different architectures is compared, namely GPU and multi-core CPU. As opposed to the widely usage of GPUs for deep learning, significant advantages from using modern CPU architectures.

Imaging genetics is an approach that explores the underlying mechanisms of brain disorders such as Alzheimer's disease (AD) by analyzing the correlation between neuroimaging and genetic data. Traditional non-negative matrix factorization (NMF) algorithms are based on linear assumptions, which limits the potential of nonlinear feature extraction among multi-omics data. This study proposes a novel joint-connectivity-based deep semi-supervised non-negative matrix factorization (JCB-DSNMF) model to overcome this limitation and incorporate prior knowledge from both within and between different modalities of data. The model effectively integrates physiological constraints such as connectivity to identify regions of interest (ROI), risk genes, and risk SNP loci associated with AD patients. JCB-DSNMF outperformed other NMF-based algorithms, such as JDSNMF and NMF, in identifying and predicting biologically relevant biomarkers closely related to AD from essential modules. The accuracy of the selected features was further validated by constructing a diagnostic model with high classification accuracy, achieving an AUC value of 0.8621 on the test set. In particular, the brain region Putamen_L and the gene RALGAPB achieved AUC values of 0.903 and 0.924, respectively, highlighting the importance of these features in early AD diagnosis.

Lee and Seung proposed nonnegative matrix factorization (NMF) algorithms to decompose patterns and images for structure retrieving. The NMF algorithms have been applied to various optimization problems. However, it is difficult to prove the convergence of this class of learning algorithms. This paper presents the global minima analysis of the NMF algorithms. In the analysis, invariant set is constructed so that the non-divergence of the algorithms can be guaranteed in the set. Using the features of linear equation systems and their solutions, the fixed points and convergence properties of the update algorithms are discussed in detail. The analysis shows that, although the cost function is not convex in both A and X together, it is possible to obtain the global minima from the particular learning algorithms. For different initializations, simulations are presented to confirm the analysis results.

Spectral un mixing has been an important technique for hyperspectral imagery processing. In spectral unmixing methods that are based on the original non-negative matrix factorization (NMF) model, unmixing accuracy is limited by the lack of appropriate constraints that derived from the inherent properties of hyperspectral images. It was shown that the correlation analysis can be extended for NMF-based spectral unmixing model, based on the advantage that the spatial and spectral correlation features can provide necessary constraints for NMF to overcome related problems and obtain better performance. However, many correlation constrained NMF models prefer adopting only spatial correlation, the neglect of spectral correlation may limit the accuracy and application scope. This letter presents a novel method of imposing spatial and spectral correlation constraints simultaneously on the NMF-based unmixing model by adopting Markov Random Field (MRF) and complexity pursuit respectively. The related issues, including reducing the deviation of endmembers' spatial energy distribution and substituting the predictability of signal for piecewise smoothness of spectra, were studied together. Experiments showed that the new proposed NMF model was superior to some existing NMF models with only spatial correlation constraint in terms of estimation accuracy of endmember spectra and fractional abundances.

The exponential random graph model (ERGM) is a popular model for social networks, which is known to have an intractable likelihood function. Sampling from the posterior for such a model is a long-standing problem in statistical research. We analyze the performance of the stochastic gradient Langevin dynamics (SGLD) algorithm (also known as noisy Longevin Monte Carlo) in tackling this problem, where the stochastic gradient is calculated via running a short Markov chain (the so-called inner Markov chain in this paper) at each iteration. We show that if the model size grows with the network size slowly enough, then SGLD converges to the true posterior in 2-Wasserstein distance as the network size and iteration number become large regardless of the length of the inner Markov chain performed at each iteration. Our study provides a scalable algorithm for analyzing large-scale social networks with possibly high-dimensional ERGMs.

In this paper, a unified Bayesian max-margin discriminant projection framework is proposed, which is able to jointly learn the discriminant feature space and the max-margin classifier with different relationships between the latent representations and observations. We assume that the latent representation follows a normal distribution whose sufficient statistics are functions of the observations. The function can be flexibly realized through either shallow or deep structures. The shallow structure includes linear, nonlinear kernel-based functions, and even the convolutional projection, which can be further trained layerwisely to build a multilayered convolutional feature learning model. To take the advantage of the deep neural networks, especially their highly expressive ability and efficient parameter learning, we integrate Bayesian modeling and the popular neural networks, for example, mltilayer perceptron and convolutional neural network, to build an end-to-end Bayesian deep discriminant projection under the proposed framework, which degenerated into the existing shallow linear or convolutional projection with the single-layer structure. Moreover, efficient scalable inferences for the realizations with different functions are derived to handle large-scale data via a stochastic gradient Markov chain Monte Carlo. Finally, we demonstrate the effectiveness and efficiency of the proposed models by the experiments on real-world data, including four image benchmarks (MNIST, CIFAR-10, STL-10, and SVHN) and one measured radar high-resolution range profile dataset, with the detailed analysis about the parameters and computational complexity.

Face recognition is a challenging issue in field of multi-science, main contents of research is how to make computer have ability of face recognition face recognition technology involved in a lot, which is a key feature extraction and classification method, this paper focuses on study of related theory. Non-negative matrix factorization trapped MF) algorithm and local non-negative matrix factorization (LNMF) algorithm is a feature extraction method based on local features, has been successfully used in face recognition NMF algorithm in face recognition rate is low, although LNMF algorithm to a certain extent, improve recognition rate, but its price is to increase number of iterations. In addition, two algorithms have failed to solve good nonlinear separable problems the kernel method combined with LNMF algorithm, kernel local non-negative matrix factorization (KLNMF) algorithm, first by a nonlinear transformation of original space to high-dimensional space, making samples linearly separable, and then use LNMF algorithm to extract face features. In classification part, paper presents decision rules of classification of their own, and design based on NMF subspace classifier. DOI : http://dx.doi.org/10.11591/telkomnika.v12i3.4356 Full Text: PDF

Bayesian model updating provides a rigorous framework to account for uncertainty induced by lack of knowledge about engineering systems in their respective mathematical models through updates of the joint probability density function (PDF), the so-called posterior PDF, of the unknown model parameters. The Markov chain Monte Carlo (MCMC) methods are currently the most popular approaches for generating samples from the posterior PDF. However, these methods often found wanting when sampling from difficult distributions (e.g., high-dimensional PDFs, PDFs with flat manifolds, multimodal PDFs, and very peaked PDFs). This paper introduces a new multi-level sampling approach for Bayesian model updating, called Sequential Gauss-Newton algorithm, which is inspired by the Transitional Markov chain Monte Carlo (TMCMC) algorithm. The Sequential Gauss-Newton algorithm improves two aspects of TMCMC to make an efficient and effective MCMC algorithm for drawing samples from difficult posterior PDFs. First, the statistical efficiency of the algorithm is enhanced by use of the systematic resampling scheme. Second, a new MCMC algorithm, called Gauss-Newton MCMC algorithm, is proposed which is essentially an M-H algorithm with a Gaussian proposal PDF tailored to the posterior PDF using the gradient and Hessian information of the negative log posterior. The effectiveness of the proposed algorithm for solving the Bayesian model updating problem is illustrated using three examples with irregularly shaped posterior PDFs.

Nonnegative matrix factorization (NMF) is a widely-used method for multivariate analysis of nonnegative data, the goal of which is decompose a data matrix into a basis matrix and an encoding variable matrix with all of these matrices allowed to have only nonnegative elements. In this paper we present simple algorithms for orthogonal NMF, where orthogonality constraints are imposed on basis matrix or encoding matrix. We develop multiplicative updates directly from the true gradient (natural gradient) in Stiefel manifold, whereas existing algorithms consider additive orthogonality constraints. Numerical experiments on face image data for a image representation task show that our orthogonal NMF algorithm preserves the orthogonality, while the goodness-of-fit (GOF) is minimized. We also apply our orthogonal NMF to a clustering task, showing that it works better than the original NMF, which is confirmed by experiments on several UCI repository data sets.

Non-negative matrix factorization (NMF) approximates a given matrix as a product of two non-negative matrix factors. Multiplicative algorithms deliver reliable results, but they show slow convergence for high-dimensional data and may be stuck away from local minima. Gradient descent methods have better behavior, but only apply to smooth losses. For non-smooth losses such as the Kullback-Leibler (KL) loss, surprisingly, these methods are lacking. In this paper, we propose a first-order primal-dual algorithm for non-negative decomposition problems (one of the two factors is fixed) with the KL distance. All required computations may be obtained in closed form and we provide an efficient heuristic way to select step-sizes. By using alternating optimization, our algorithm readily extends to NMF and, on synthetic or real world data, it is either faster than existing algorithms, or leads to improved local optima, or both.

Hyperspectral unmixing is a powerful method of the remote sensing image mining that identifies the constituent materials and estimates the corresponding fractions from the mixture. We consider the application of nonnegative matrix factorization (NMF) for the mining and analysis of spectral data. In this paper, we develop two effective active set type NMF algorithms for hyperspectral unmixing. Because the factor matrices used in unmixing have sparse features, the active set strategy helps reduce the computational cost. These active set type algorithms for NMF is based on an alternating nonnegative constrained least squares (ANLS) and achieve a quadratic convergence rate under the reasonable assumptions. Finally, numerical tests demonstrate that these algorithms work well and that the function values decrease faster than those obtained with other algorithms.

ABSTRACTHyperspectral unmixing (HU) has drawn remarkable attention because it can decompose mixed pixels into a set of endmembers and abundance fractions. And the nonnegative matrix factorization (NMF) algorithm has been widely used for solving hyperspectral spectral unmixing problem. This letter proposes a Hessian graph regularized NMF (HGNMF) algorithm, which relies upon the construction of a Hessian graph representation, to solve the hyperspectral unmixing problem. According to the HGNMF algorithm, the smoothness in the estimated abundance maps is well promoted. Moreover, the optimized problem of HGNMF algorithm is also solved by employing the multiplicative updating rules. Compared to other algorithms, the proposed HGNMF algorithm demonstrates lots of advantages based on the simulated results of the synthetic data and real data sets.

One of the most common types of models that helps us to understand neuron behavior is based on the Hodgkin–Huxley ion channel formulation (HH model). A major challenge with inferring parameters in HH models is non-uniqueness: many different sets of ion channel parameter values produce similar outputs for the same input stimulus. Such phenomena result in an objective function that exhibits multiple modes (i.e., multiple local minima). This non-uniqueness of local optimality poses challenges for parameter estimation with many algorithmic optimization techniques. HH models additionally have severe non-linearities resulting in further challenges for inferring parameters in an algorithmic fashion. To address these challenges with a tractable method in high-dimensional parameter spaces, we propose using a particular Markov chain Monte Carlo (MCMC) algorithm, which has the advantage of inferring parameters in a Bayesian framework. The Bayesian approach is designed to be suitable for multimodal solutions to inverse problems. We introduce and demonstrate the method using a three-channel HH model. We then focus on the inference of nine parameters in an eight-channel HH model, which we analyze in detail. We explore how the MCMC algorithm can uncover complex relationships between inferred parameters using five injected current levels. The MCMC method provides as a result a nine-dimensional posterior distribution, which we analyze visually with solution maps or landscapes of the possible parameter sets. The visualized solution maps show new complex structures of the multimodal posteriors, and they allow for selection of locally and globally optimal value sets, and they visually expose parameter sensitivities and regions of higher model robustness. We envision these solution maps as enabling experimentalists to improve the design of future experiments, increase scientific productivity and improve on model structure and ideation when the MCMC algorithm is applied to experimental data.