A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

Non-negative matrix factorization (NMF) has been one of the most popular methods for feature learning in the field of machine learning and computer vision. Most existing works directly apply NMF on high-dimensional image datasets for computing the effective representation of the raw images. However, in fact, the common essential information of a given class of images is hidden in their low rank parts. For obtaining an effective low-rank data representation, we in this paper propose a non-negative low-rank matrix factorization (NLMF) method for image clustering. For the purpose of improving its robustness for the data in a manifold structure, we further propose a graph regularized NLMF by incorporating the manifold structure information into our proposed objective function. Finally, we develop an efficient alternating iterative algorithm to learn the low-dimensional representation of low-rank parts of images for clustering. Alternatively, we also incorporate robust principal component analysis into our proposed scheme. Experimental results on four image datasets reveal that our proposed methods outperform four representative methods.

Nonnegative Matrix Factorization (NMF) is a widely used technique in many applications such as face recognition, motion segmentation, etc. It approximates the nonnegative data in an original high dimensional space with a linear representation in a low dimensional space by using the product of two nonnegative matrices. In many applications data are often partially corrupted with large additive noise. When the positions of noise are known, some existing variants of N-MF can be applied by treating these corrupted entries as missing values. However, the positions are often unknown in many real world applications, which prevents the usage of traditional NMF or other existing variants of NMF. This paper proposes a Robust Nonnegative Matrix Factorization (RobustNMF) algorithm that explicitly models the partial corruption as large additive noise without requiring the information of positions of noise. In particular, the proposed method jointly approximates the clean data matrix with the product of two nonnegative matrices and estimates the positions and values of outliers/noise. An efficient iterative optimization algorithm with a solid theoretical justification has been proposed to learn the desired matrix factorization. Experimental results demonstrate the advantages of the proposed algorithm.

In this paper, we present a novel muscle synergy extraction method based on multivariate curve resolution–alternating least squares (MCR-ALS) to overcome the limitation of the nonnegative matrix factorization (NMF) method for extracting non-sparse muscle synergy, and we study its potential application for evaluating motor function of stroke survivors. Nonnegative matrix factorization (NMF) is the most widely used method for muscle synergy extraction. However, NMF is susceptible to components’ sparseness and usually provides inferior reliability, which significantly limits the promotion of muscle synergy. In this study, MCR-ALS was employed to extract muscle synergy from electromyography (EMG) data. Its performance was compared with two other matrix factorization algorithms, NMF and self-modeling mixture analysis (SMMA). Simulated data sets were utilized to explore the influences of the sparseness and noise on the extracted synergies. As a result, the synergies estimated by MCR-ALS were the most similar to true synergies as compared with SMMA and NMF. MCR-ALS was used to analyze the muscle synergy characteristics of upper limb movements performed by healthy (n = 11) and stroke (n = 5) subjects. The repeatability and intra-subject consistency were used to evaluate the performance of MCR-ALS. As a result, MCR-ALS provided much higher repeatability and intra-subject consistency as compared with NMF, which were important for the reliability of the motor function evaluation. The stroke subjects had lower intra-subject consistency and seemingly had more synergies as compared with the healthy subjects. Thus, MCR-ALS is a promising muscle synergy analysis method for motor function evaluation of stroke patients.

Positron emission tomography (PET) is an important imaging technique to visualize a number of functions in the brain or human body. For reconstructing PET images from the sinogram data, an inverse problem has to be solved using numerical optimizations such as expectation-maximization (EM)-based methods. However, the standard EM method suffers from measurement noise added in the sinogram data. In this paper, we propose a new simultaneous PET image reconstruction and parts extraction method using constrained non-negative matrix factorization. In contrast that the many existing methods reconstruct a single PET image independently, we reconstruct the time-series of PET images simultaneously from the time-series of sinograms using non-negative matrix factorization. Furthermore, we impose the smoothness constraint for the temporal feature, and the exclusive LASSO-based sparseness constraint for the spatial feature for robust image reconstruction and physically meaningful feature extraction.

Guitar string separation is a novel and complicated task. Guitar notes are not pure steady-state signals, hence, we hypothesize that neither Non-Negative Matrix Factorization (NMF) nor Non-Negative Matrix Factor Deconvolution (NMFD) are optimal for separating them. Therefore, we separate steady-state and transient parts using Harmonic-Percussive Separation (HPS) as a preprocessing step. Then, we use NMF for factorizing the harmonic part and NMFD for deconvolving the percussive part. We make use of a hexaphonic guitar dataset which allows for objective evaluation. In addition, we compare several types of time-frequency mask and introduce an intuitive way to combine a binary mask with a ratio mask. We show that the HPS mask type has an effect on source estimation. Our proposed method achieved results comparable to NMF without HPS. Finally, we show that the optimal mask at the final separation stage depends on the estimation algorithm.

Musculoskeletal models (MMs) driven by electromyography (EMG) signals have been used to predict human movements. Muscle excitations of MMs are generally the amplitude of EMG, which shows large variability even when repeating the same task. The general structure of muscle synergies has been proved to be consistent across test sessions, providing a perspective for extracting stable control information for MMs. Although non-negative matrix factorization (NMF) is a common method for extracting synergies, the factorization result of NMF is not unique. In this study, we proposed an improved NMF algorithm for extracting stable control information of MMs to predict hand and wrist motions. Specifically, we supplemented the Hadamard product and L2-norm regularization term to the objective function of NMF. The proposed NMF was utilized to identify stable muscle synergies. Then, the time-varying profile of each synergy was fed into a subject-specific MM for estimating joint motions. The results demonstrated that the proposed scheme significantly outperformed a traditional MM and an MM combined with the classic NMF (NMF-MM), with averaged R and NRMSE equal to \(0.89\pm 0.06\) and \(0.16\pm 0.04\). Further, the similarity between muscle synergies extracted from different training data revealed the proposed method’s effectiveness of identifying consistent control information for MMs. This study provides a novel model-based scheme for the estimation of continuous movements.KeywordsMusculoskeletal modelElectromyographyNon-negative matrix factorization algorithm

Non-negative Matrix Factorization (NMF) has been frequently applied to image feature extraction and clustering. Especially in image clustering tasks, it can achieve the similar or better performance than most of the matrix factorization algorithms due to its parts-based representations in the brain. However, the features extracted by NMF are not sparse and localized enough and the error of factorization is not small enough. Semi-tensor product of matrices (STP) is a novel operation of matrix multiplication, it is a generalization of the conventional matrix product for allowing the dimensions of factor matrices to be unequal. STP can manage the data hierarchically and the inverse process of STP can separate the data hierarchically. Based on this character of STP, we propose the Non-Negative Matrix Semi-Tensor Factorization (NMSTF). In this algorithm, we use the inverse process of Semi-Tensor Product of matrices for non-negative matrix factorization. This algorithm effectively optimizes the above two problems in NMF. While achieving similar even better performance on image clustering tasks, the size of features extracted by STNMF is at least 50 % smaller than the ones’ extracted by NMF and the error of factorization reduces 30 % in average.

The Positive Matrix Factorisation (PMF) algorithm (Paatero and Tapper, 1994) has been the most widely used receptor model for a long time and has only recently been challenged with new methodologies. The novel Bayesian auto-correlated matrix factorisation method (BAMF, Rusanen et al. 2024) integrates an auto-correlation term emulating real-world pollutant sources time evolution has produced higher accuracy compared to PMF. However, both PMF and BAMF struggle to provide well-separated profiles manifested as mixed time series contributions. A sparsity-handling algorithm named horseshoe (HS) regularisation has beenapplied to BAMF in order to improve profile determination. The horseshoe application pushes some parameters to be close to zero and others to have large values (Piironen and Vehtari, 2017). The BAMF+HS method reduces the dimensionality of the problem by suppressing the non-significant species for each profile. The resulting profiles are expected to be less noisy and better representing the nature of the atmospheric pollution sources. Figure 1 shows the effect of BAMF+HS (in orange) compared to the regular BAMF (in blue) and the PMF (in green) on a toy dataset, consisting on an oversimplified dataset with very sparse profiles. The BAMF+HS results show contributions pushed to zero, making the profiles closer to the truth (in black) with respect to the less sparse results of BAMF and PMF. This same comparison has been carried out on realistic synthetic datasets to show the effectiveness of sparsity introduction into source apportionment.Figure 1. Comparison to truth of source apportionment profiles resulting from three different receptor models for a toy dataset.Acknowledgement: This work is supported by the European Union's Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Postdoctoral Fellowship Programme, SMASH co-funded under the grant agreement No. 101081355. The SMASH project is co-funded by the Republic of Slovenia and the European Union from the European Regional Development Fund. K.R.D. acknowledges support by SNSF Ambizione grant PZPGP2_201992.ReferencesPiironen, J., & Vehtari, A. (2017). Sparsity information and regularization in the horseshoe and other shrinkage priors.Rusanen, A., Björklund, A., Manousakas, M., Jiang, J., Kulmala, M. T., Puolamäki, K., & Daellenbach, K. R. (2023). Atmospheric Measurement Techniques Discussions, 2023, 1-28.Paatero, P., & Tapper, U. (1994). Environmetrics, 5(2), 111-126.

Abstract. A new methodology for performing long-term source apportionment (SA) using positive matrix factorization (PMF) is presented. The method is implemented within the SoFi Pro software package and uses the multilinear engine (ME-2) as a PMF solver. The technique is applied to a 1-year aerosol chemical speciation monitor (ACSM) dataset from downtown Zurich, Switzerland. The measured organic aerosol mass spectra were analyzed by PMF using a small (14 d) and rolling PMF window to account for the temporal evolution of the sources. The rotational ambiguity is explored and the uncertainties of the PMF solutions were estimated. Factor–tracer correlations for averaged seasonal results from the rolling window analysis are higher than those retrieved from conventional PMF analyses of individual seasons, highlighting the improved performance of the rolling window algorithm for long-term data. In this study four to five factors were tested for every PMF window. Factor profiles for primary organic aerosol from traffic (HOA), cooking (COA) and biomass burning (BBOA) were constrained. Secondary organic aerosol was represented by either the combination of semi-volatile and low-volatility organic aerosol (SV-OOA and LV-OOA, respectively) or by a single OOA when this separation was not robust. This scheme led to roughly 40 000 PMF runs. Full visual inspection of all these PMF runs is unrealistic and is replaced by predefined user-selected criteria, which allow factor sorting and PMF run acceptance/rejection. The selected criteria for traffic (HOA) and BBOA were the correlation with equivalent black carbon from traffic (eBCtr) and the explained variation of m/z 60, respectively. COA was assessed by the prominence of a lunchtime concentration peak within the diurnal cycle. SV-OOA and LV-OOA were evaluated based on the fractions of m/z 43 and 44 in their respective factor profiles. Seasonal pre-tests revealed a non-continuous separation of OOA into SV-OOA and LV-OOA, in particular during the warm seasons. Therefore, a differentiation between four-factor solutions (HOA, COA, BBOA and OOA) and five-factor solutions (HOA, COA, BBOA, SV-OOA and LV-OOA) was also conducted based on the criterion for SV-OOA. HOA and COA contribute between 0.4–0.7 µg m−3 (7.8 %–9.0 %) and 0.7–1.2 µg m−3 (12.2 %–15.7 %) on average throughout the year, respectively. BBOA shows a strong yearly cycle with the lowest mean concentrations in summer (0.6 µg m−3, 12.0 %), slightly higher mean concentrations during spring and fall (1.0 and 1.5 µg m−3, or 15.6 % and 18.6 %, respectively), and the highest mean concentrations during winter (1.9 µg m−3, 25.0 %). In summer, OOA is separated into SV-OOA and LV-OOA, with mean concentrations of 1.4 µg m−3 (26.5 %) and 2.2 µg m−3 (40.3 %), respectively. For the remaining seasons the seasonal concentrations of SV-OOA, LV-OOA and OOA range from 0.3 to 1.1 µg m−3 (3.4 %–15.9 %), from 0.6 to 2.2 µg m−3 (7.7 %–33.7 %) and from 0.9 to 3.1 µg m−3 (13.7 %–39.9 %), respectively. The relative PMF errors modeled for this study for HOA, COA, BBOA, LV-OOA, SV-OOA and OOA are on average ±34 %, ±27 %, ±30 %, ±11 %, ±25 % and ±12 %, respectively.

In this letter, we consider the nonnegative matrix factorization (NMF) problem and several NMF variants. Two approaches based on DC (difference of convex functions) programming and DCA (DC algorithm) are developed. The first approach follows the alternating framework that requires solving, at each iteration, two nonnegativity-constrained least squares subproblems for which DCA-based schemes are investigated. The convergence property of the proposed algorithm is carefully studied. We show that with suitable DC decompositions, our algorithm generates most of the standard methods for the NMF problem. The second approach directly applies DCA on the whole NMF problem. Two algorithms-one computing all variables and one deploying a variable selection strategy-are proposed. The proposed methods are then adapted to solve various NMF variants, including the nonnegative factorization, the smooth regularization NMF, the sparse regularization NMF, the multilayer NMF, the convex/convex-hull NMF, and the symmetric NMF. We also show that our algorithms include several existing methods for these NMF variants as special versions. The efficiency of the proposed approaches is empirically demonstrated on both real-world and synthetic data sets. It turns out that our algorithms compete favorably with five state-of-the-art alternating nonnegative least squares algorithms.

The significance of Non-negative Matrix Factorization ((NMF) in the field of automatic text summarization is rapidly increasing due to its interpretation and storage capabilities. Interpretation defines the ease at which the structure of high dimensional data can be understood. While, storage capability relates to the extent of data reduction process achieved by NMF. The parametric values that serve as input to the NMF process include initialization method, rank of factorization, sparseness measure and maximum iteration. These inputs are vital to the output produced by NMF. These parameters of NMF were not been considered in the existing literature on Automatic Text Summarization (ATS). This paper sheds light specifically on sparseness of NMF when applied to ATS and the impact it makes to the quality of the summary generated. In this study on NMF algorithm that supports sparseness is treated with various degree of sparsity and its performance impact on text summary generated is compared with other NMF algorithms without sparseness constraints.

Positive Matrix Factorization (PMF) is a powerful method for the apportionment of aerosol sources. Traditionally applied to chemical datasets, it has more recently been extended to physical datasets, focusing predominantly on size distributions of ultra-fine particles (Hopke et al., 2022). In this study, aerosol physical properties (particle size distributions, from ultrafine to coarse mode, and spectral aerosol absorption) were used as input to PMF (EPA 5.0) for identifying emission sources of two distinct sites, and results compared with those from chemical PMF. Located at the two extremes of the Italian territory, the sites of Aosta and Lecce represent two markedly different environments. The first lies in the mountainous area of the northwestern Alps, the second in a flat area at the southeastern edge of the peninsula, in a typical Mediterranean context. In previous studies, PMF of aerosol chemical composition was performed using chemical speciation of PM over the period 2019-2021 (Aosta, e.g. Diemoz et al., 2019) and 2017 (Lecce, Giannossa et al., 2022).  The PMF input dataset at the urban-background site in Aosta included aerosol size distributions measured by an Optical Particle Counter (OPC) across the 0.18–18 µm range and wavelength-dependent aerosol absorption from an AE33-aethalometer. At the urban-background observatory in Lecce, an aerosol size distribution range (0.02–10 µm) was available using both a Scanning Mobility Particle Sizer (SMPS) and an OPC, complemented by single-wavelength aerosol absorption (Multi-Angle Absorption Photometer, MAAP) providing eBC concentrations. Ancillary data included meteorological parameters and trace-gases concentration. Although with differences in the physical properties used, at both sites the physical PMF allowed the identification of 6 aerosol sources. In Aosta, the aerosol sources included fossil fuel combustion, biomass burning, secondary droplet mode, secondary condensation, dust, and coarse particles. In Lecce, the sources were nucleation (thanks to the additional use of the SMPS), traffic emissions, secondary nitrate, secondary sulfate, regional transport (sea spray and dust), and local resuspension (coarse particles). The work will present a comparison of the two PMF approaches (Chemistry- and Physics-based PMF) and advantages and limits of the different physical input datasets used at the two sites. For example, at the Lecce site, the information on ultrafine and fine particle distribution well captured features of the nucleation and traffic factors, while the absence of wavelength-dependent absorption coefficients limited the ability to distinguish biomass burning from fossil fuel sources, which was key for Aosta. Overall, the use of physical aerosol data as input to PMF proved to be an effective method for source apportionment and could usefully complement the chemical-PMF analysis. In fact, this approach offers significant advantages, such as the capability for quasi real-time monitoring and the relative ease of instruments use and data analysis compared to the ‘traditional’ chemical analysis of PM. The authors acknowledge the MUR for funding the research through the CIR01_00015-PER-ACTRIS-IT.  Diémoz, H.et al., https://doi.org/10.5194/acp-19-10129-2019, 2019.  Giannossa, L. et al., http://doi.10.1016/j.jenvman.2022.115752, 2022. Hopke, P. K. et al., https://doi.org/10.1016/j.scitotenv.2022.153104, 2022. 

Low-rank matrix factorizations such as Principal Component Analysis (PCA), Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF) are a large class of methods for pursuing the low-rank approximation of a given data matrix. The conventional factorization models are based on the assumption that the data matrices are contaminated stochastically by some type of noise. Thus the point estimations of low-rank components can be obtained by Maximum Likelihood (ML) estimation or Maximum a posteriori (MAP). In the past decade, a variety of probabilistic models of low-rank matrix factorizations have emerged. The most significant difference between low-rank matrix factorizations and their corresponding probabilistic models is that the latter treat the low-rank components as random variables. This paper makes a survey of the probabilistic models of low-rank matrix factorizations. Firstly, we review some probability distributions commonly-used in probabilistic models of low-rank matrix factorizations and introduce the conjugate priors of some probability distributions to simplify the Bayesian inference. Then we provide two main inference methods for probabilistic low-rank matrix factorizations, i.e., Gibbs sampling and variational Bayesian inference. Next, we classify roughly the important probabilistic models of low-rank matrix factorizations into several categories and review them respectively. The categories are performed via different matrix factorizations formulations, which mainly include PCA, matrix factorizations, robust PCA, NMF and tensor factorizations. Finally, we discuss the research issues needed to be studied in the future.

In this work, we will consider the dimension reduction of the set of time series, such as economic data, to find the meaningful basis vector for the set of data, and indicate which data use which basis vector. Usually each of the time series is analyzed independently in economics but here we will analyze the set of time series simultaneously. Since some of the economic data are measured as positive values and we want to decompose them as a mixture of the parts, we will apply non-negative matrix factorization to the economic data. Non-negative matrix factorization can compress dimensions by approximating a non-negative matrix with the product of two non-negative matrices. The two non-negative matrices are called the coefficient matrix and the basis matrix, and the basis matrix can be considered as a dimensionally compressed matrix. If the standard non-negative matrix factorization is used for economic data, the basis matrix may not be smooth. We think that the basis vectors should be smooth except a few special economical incidents. In the proposed method, a Graph-based non-negative matrix factorization is introduced to regularize the basis matrix of the time series. A path graph for representing the time series of economic data is incorporated into the non-negative matrix factorization as regularization. As a result, basis vectors that maintains the time series of economic data are decomposed. Furthermore, we propose to introduce a sparsity in the non-negative matrix factorization. Traditionally, the sparsity incorporated into non-negative matrix factorization has been used for basis vectors. However, the proposed method introduces the sparsity for coefficient vectors. Thus the proposed method, which simultaneously incorporates the sparsity for the coefficient vectors and the smoothness for the basis vectors, can extract the smooth basis vectors and the original economical data are approximated as the weighted sum of the few bases vectors. This allows us to discover economic trends and the best-fit trends for each data at the same time.

As an important matrix factorization model, Nonnegative Matrix Factorization (NMF) has been widely used in information retrieval and data mining research. Standard Nonnegative Matrix Factorization is known to use the Frobenius norm to calculate the residual, making it sensitive to noises and outliers. It is desirable to use robust NMF models for practical applications, in which usually there are many data outliers. It has been studied that the 2,1, or 1-norm can be used for robust NMF formulations to deal with data outliers. However, these alternatives still suffer from the extreme data outliers. In this paper, we present a novel robust capped norm orthogonal Nonnegative Matrix Factorization model, which utilizes the capped norm for the objective to handle these extreme outliers. Meanwhile, we derive a new efficient optimization algorithm to solve the proposed non-convex non-smooth objective. Extensive experiments on both synthetic and real datasets show our proposed new robust NMF method consistently outperforms related approaches.