Non-negative matrix factorization algorithms generally improve topic model fits.

Topic modeling extracts useful potential topics that reflect market information from massive financial news and is widely used in data mining and economic research. Traditional topic modeling approaches such as Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) lack semantic information, and short texts have feature sparse problems. We develop a topic clustering model based on BERT-LDA joint embedding that takes both contextual semantics and thematic narrative into account. We cluster document embeddings with the HDBSCAN algorithm and utilize a class-based TF-IDF (c-TF-IDF) method to create topic representations. Empirical results show that the BERT-LDA model is competitive compared with traditional and single topic models. It generates coherent topic words that are dissimilar to each other.

BackgroundTopic models are a class of unsupervised machine learning models, which facilitate summarization, browsing and retrieval from large unstructured document collections. This study reviews several methods for assessing the quality of unsupervised topic models estimated using non-negative matrix factorization. Techniques for topic model validation have been developed across disparate fields. We synthesize this literature, discuss the advantages and disadvantages of different techniques for topic model validation, and illustrate their usefulness for guiding model selection on a large clinical text corpus.Design, setting and dataUsing a retrospective cohort design, we curated a text corpus containing 382,666 clinical notes collected between 01/01/2017 through 12/31/2020 from primary care electronic medical records in Toronto Canada.MethodsSeveral topic model quality metrics have been proposed to assess different aspects of model fit. We explored the following metrics: reconstruction error, topic coherence, rank biased overlap, Kendall’s weighted tau, partition coefficient, partition entropy and the Xie-Beni statistic. Depending on context, cross-validation and/or bootstrap stability analysis were used to estimate these metrics on our corpus.ResultsCross-validated reconstruction error favored large topic models (K ≥ 100 topics) on our corpus. Stability analysis using topic coherence and the Xie-Beni statistic also favored large models (K = 100 topics). Rank biased overlap and Kendall’s weighted tau favored small models (K = 5 topics). Few model evaluation metrics suggested mid-sized topic models (25 ≤ K ≤ 75) as being optimal. However, human judgement suggested that mid-sized topic models produced expressive low-dimensional summarizations of the corpus.ConclusionsTopic model quality indices are transparent quantitative tools for guiding model selection and evaluation. Our empirical illustration demonstrated that different topic model quality indices favor models of different complexity; and may not select models aligning with human judgment. This suggests that different metrics capture different aspects of model goodness of fit. A combination of topic model quality indices, coupled with human validation, may be useful in appraising unsupervised topic models.

Topic modeling, a pivotal unsupervised machine learning approach, serves as a valuable tool for uncovering latent themes within vast document repositories. It aids in the organization, comprehension, and simplification of extensive textual data while revealing distinctive underlying themes across a corpus of documents. The intrinsic characteristics of social media content, marked by brevity, text-heavy nature, and a lack of structure, often pose methodological challenges in data collection and analysis. In an effort to bridge the realms of computer science and empirical social sciences, this research aims to assess the effectiveness of three distinct topic modeling methodologies: Bidirectional Encoder Representations from Transformers (BERTopic), Non-negative Matrix Factorization (NMF), and Latent Dirichlet Allocation (LDA). While NMF relies on a matrix factorization paradigm and LDA employs a probabilistic framework, BERT-based techniques, which utilize sentence embeddings for topic generation, represent a contemporary innovation. In this study, BERTopic is evaluated with multiple pre-trained sentence embeddings, and the outcomes are rigorously compared with those derived from LDA and NMF methodologies. The study leverages C_V and U_MASS, two vital coherence measures, to evaluate the efficacy of these topic modeling strategies. The research delves into the analysis of various algorithms, elucidating their strengths and limitations within the context of social sciences, using YouTube comments as a benchmark dataset. Notably, this investigation sheds light on the utility of BERTopic and NMF for evaluating YouTube video content disclosure based on specific attributes, thereby enhancing the analysis process and addressing performance concerns.

An analysis of the coherence of descriptors in topic modeling

Text embedding plays a crucial role in natural language processing (NLP). Among various approaches, nonnegative matrix factorization (NMF) is an effective method for this purpose. However, the standard NMF approach, fundamentally based on the bag-of-words model, fails to utilize the contextual information of documents and may result in a significant loss of semantics. To address this issue, we propose a new NMF scheme incorporating a regularization term based on the Wasserstein metric, which quantifies an approximation of a word-context matrix to leverage semantic information. Since the word-context matrix is typically symmetric and positive definite (SPD), the Wasserstein metric used in the regularization is related to a natural SPD manifold structure. We then build upon this manifold structure to integrate an explicit expression of the Wasserstein metric into the NMF framework. Additionally, we introduce modifications to the gradient computation to adapt three fundamental classes of numerical algorithms from ordinary NMF to tackle the Wasserstein-regularized NMF (NMF-WR) problem. To demonstrate the effectiveness of the NMF-WR model and the proposed algorithms, we conduct experiments on popular datasets, focusing on topic modeling and document clustering. The results indicate that the NMF-WR model exhibits superior performance compared to other conventional NMF models in processing text embeddings. These findings suggest that this novel NMF-WR framework not only enhances semantic representation but also underscores a commitment to the model’s reliability and interpretability.

This project explores the application of sophisticated topic models to determine the ideal amount of subjects and evaluate them across several datasets pertaining to trends and current events. We propose a comprehensive approach that leverages the topic models such as Latent Dirichlet Allocation (LDA), Non-negative Matrix Factorization (NMF), Latent Semantic Analysis (LSA), Probabilistic Latent Dirichlet Allocation (PLDA) and Probabilistic Latent Semantic Indexing (PLSI) to accurately identify andanalyse topics. Our approach utilizes various topic modelling techniques to enhance the robustness and precision of topic detection. The results demonstrate the effectiveness of topic models in capturing nuanced trends and themes within complex datasets. A user-friendly frontend, built with Flask and SQLite, enables interactive input and visualization of topic modeling results, offering an accessible platformfor real-time topic analysis and insights. Thismethodology provides a powerful tool for understanding and responding to emerging trends in dynamic datasets

This paper proposes a method for obtaining nonlinear models of distribution grid based on available measurements from the power grid. We formulate a maximum likelihood estimation (MLE) problem that estimates unknown line parameters—specifically, the impedance between nodes—using measured voltage magnitudes and net active and reactive power injections at each node. The nonlinear model for the distribution grid uses a nonlinear approximation of the DistFlow model, which includes line losses and is parameterized by the unknown line impedances. We solve the resulting MLE problem using an expectation maximization (EM) algorithm, tailored for the nonlinear model, and provide a numerically robust implementation. The proposed method is demonstrated on the IEEE 37-node test network, and we compare it with the state-of-the-art methods. The proposed method achieves a 70% reduction in voltage error and an error for state variables that is more than 10000 times smaller. A final comparison uses data from a real network, and the proposed method achieves parameter estimates with errors 100 times smaller than competing methods.

Non-negative matrix factorization (NMF) is the problem of determining two non-negative low rank factors $\mathbf{W}$ and $\mathbf{H}$ , for the given input matrix $\mathbf{A}$ , such that $\mathbf{A}\approx \mathbf{W}\mathbf{H}$ . NMF is a useful tool for many applications in different domains such as topic modeling in text mining, background separation in video analysis, and community detection in social networks. Despite its popularity in the data mining community, there is a lack of efficient parallel algorithms to solve the problem for big data sets. The main contribution of this work is a new, high-performance parallel computational framework for a broad class of NMF algorithms that iteratively solves alternating non-negative least squares (NLS) subproblems for $\mathbf{W}$ and $\mathbf{H}$ . It maintains the data and factor matrices in memory (distributed across processors), uses MPI for interprocessor communication, and, in the dense case, provably minimizes communication costs (under mild assumptions). The framework is flexible and able to leverage a variety of NMF and NLS algorithms, including Multiplicative Update, Hierarchical Alternating Least Squares, and Block Principal Pivoting. Our implementation allows us to benchmark and compare different algorithms on massive dense and sparse data matrices of size that spans from few hundreds of millions to billions. We demonstrate the scalability of our algorithm and compare it with baseline implementations, showing significant performance improvements. The code and the datasets used for conducting the experiments are available online.

Topic modeling is an effective way to gain insight into large amounts of data.Some of the most widely used topic models are Latent Dirichlet allocation (LDA) and Nonnegative Matrix Factorization (NMF).However, new ways to mine topics have emerged with the rise of self-attention models and pretrained language models.BERTopic represents the current stateof-the-art when it comes to modeling topics.In this paper, we compared LDA, NMF, and BERTopic performance on literary texts in the Serbian language, both quantitatively by measuring Topic Coherency (TC) and Topic Diversity (TD), and by conducting a qualitative evaluation of the obtained topics.Additionally, for BERTopic, we compared multilingual sentence transformer embeddings with the Jerteh-355 monolingual embeddings for Serbian.NMF yielded the best Topic Coherency results, while BERTopic with Jerteh-355 embeddings gave the best Topic Diveristy.The monolingual Serbian Jerteh-355 embeddings also outperformed sentence transformer embeddings in both TC and TD.

In this paper, we propose a document topic model (DTM) based on the non-negative matrix factorization (NMF) approach to explore spontaneous spoken document retrieval. The model uses latent semantic indexing to detect underlying semantic relationships within documents. Each document is interpreted as a generative topic model belonging to many topics. The relevance of a document to a query is expressed by the probability of a query being generated by the model. The term-document matrix used for NMF is built stochastically from the speech recognition N-best results, so that multiple recognition hypotheses can be utilized to compensate for the word recognition errors. Using this approach, experiments are conducted on a test collection from the Corpus of Spontaneous Japanese (CSJ), with 39 queries for over 600 hours of spontaneous Japanese speech. The retrieval performance of this model is proved to be superior to the conventional vector space model (VSM) when the dimension or topic number exceeds a certain threshold. Moreover, whether from the viewpoint of retrieval performance or the ability of topic expression, the NMF-based topic model is verified to surpass another latent indexing method that is based on the singular value decomposition (SVD). The extent to which this topic model can resist speech recognition error, which is a special problem of spoken document retrieval, is also investigated.

Objective: The objective is to develop an ensemble of machine learning algorithms to identify multilingual, online articles that are relevant to biosurveillance. Language morphology varies widely across languages and must be accounted for when designing algorithms. Here, we compare the performance of a word embedding-based approach and a topic modeling approach with machine learning algorithms to determine the best method for Chinese, Arabic, and French languages.Introduction: Global biosurveillance is an extremely important, yet challenging task. One form of global biosurveillance comes from harvesting open source online data (e.g. news, blogs, reports, RSS feeds). The information derived from this data can be used for timely detection and identification of biological threats all over the world. However, the more inclusive the data harvesting procedure is to ensure that all potentially relevant articles are collected, the more data that is irrelevant also gets harvested. This issue can become even more complex when the online data is in a non-native language. Foreign language articles not only create language-specific issues for Natural Language Processing (NLP), but also add a significant amount of translation costs. Previous work shows success in the use of combinatory monolingual classifiers in specific applications, e.g., legal domain [1]. A critical component for a comprehensive, online harvesting biosurveillance system is the capability to identify relevant foreign language articles from irrelevant ones based on the initial article information collected, without the additional cost of full text retrieval and translation.Methods: The analysis text dataset contains the title and brief description of 3506 online articles in Chinese, Arabic, and French languages from the date range of August, 17, 2016 to July 5, 2017. The NLP article pre-processing steps are language-specific tokenization and stop words removal. We compare two different approaches: word embeddings and topic modeling (Fig. 1). For word embeddings, we first generate word vectors for the data using a pretrained Word2Vec (W2V) model [2]. Subsequently, the word vectors within a document are averaged to produce a single feature vector for the document. Then, we fit a machine learning algorithm (random forest classifier or Support Vector Machine (SVM)) to the training vectors and get predictions for the test documents. For topic modelling, we used a Latent Dirichlet Allocation (LDA) model to generate five topics for all relevant documents [3]. For each new document, the output is the probability measure for the document belonging to these five topics. Here, we classify the new document by comparing the probability measure with a relevancy threshold.Results: The Word2Vec model combined with a random forest classifier outperformed the other approaches across the three languages (Fig. 2); the Chinese model has an 89% F1-score, the Arabic model has 86%, and the French model has 94%. To decrease the chance of calling a potentially relevant article irrelevant, high recall was more important than high precision. In the Chinese model, the Word2Vec with a random forest approach had the highest recall at 98% (Table 1).Conclusions: We present research findings on different approaches of relevance to biosurveillance identification on non-English texts and identify the best performing methods for implementation into a biosurveillance online article harvesting system. Our initial results suggest that the word embeddings model has an advantage over topic modeling, and the random forest classifier outperforms the SVM. Directions for future work will aim to further expand the list of languages and methods to be compared, e.g., n-grams and non-negative matrix factorization. In addition, we will fine-tune the Arabic and French model for better accuracy results.

Along with the exponential growth of text data on the Web, particularly of the user-generated content, comes an increasing need for hierarchically organizing documents, retrieving documents accurately, and discovering evolutionary trends of various popular topics from the data. However, all of these are challenging due to the diversity, heterogeneity, noisiness and time-sensitivity of Web 2.0 data. Motivated by this, we tackle the challenges at a fundamental level, by proposing a novel topic modeling method with ontological guidance. It may be used to discover topic language models formalizing various terms relevant to given topics using the Web data. The topic model takes into account both the ontological relationships amongst the topics defined in a topic taxonomy and also word co-occurrence patterns in the data to automatically identify the portions in the data relevant to the topics. Then, it estimates language models for these topics from these relevant portions. At an application level, we use the topic model to propose novel approaches for three different tasks, namely hierarchical text classification without labeled data, information retrieval with pseudo-relevance feedback, and discovering topic evolutionary trends. Our classification experiment on the IPTC (International Press and Telecommunications Council) taxonomy, containing more than 1100 topics, shows that our approach achieves a performance of 67% in terms of the hierarchical version of the F-1 measure, without using any labeled data. Our retrieval experiments on five benchmark datasets show that compared to baseline retrieval (without pseudo-relevance feed-back), our approach improves on average 39% in terms of mean average precision. Finally, for the last task, using blog data, our approach discovers meaningful insights on how the crowd responds to various news topics such as the language used to discuss each topic, how this language drifts over time, and when the crowd’s focus on a topic increases, reaches a peak, and declines.

Meta-learning is originally developed for supervised learning to enable models to generalize across tasks by leveraging prior experience. However, its potential in the unsupervised learning domain remains largely underexplored. To demonstrate the feasibility of meta-learning in unsupervised settings, we apply it to Non-negative Matrix Factorization (NMF), a widely used technique for decomposing non-negative data matrices into interpretable, lower-dimensional representations. While NMF has found applications in topic modeling, image processing, bioinformatics, and recommendation systems, it faces persistent challenges such as rank selection, optimization stability, uniqueness, and computational efficiency. Traditional approaches primarily focus on improving initialization strategies to enhance convergence and generalization. However, these methods often fail to exploit structural similarities across tasks. In this paper, we introduce a meta-learning paradigm for NMF that systematically learns optimal factorization parameters from small-scale tasks and transfers this knowledge to improve learning on larger tasks. By discovering fine structures in small tasks and leveraging them to guide factorization on more complex datasets, our approach directs the search process toward a more optimal and structured search space, reducing the risk of suboptimal solutions and improving model robustness. This meta-unsupervised learning framework enhances NMF's ability to uncover meaningful patterns while maintaining adaptability across different domains. Additionally, we evaluate the model under noisy conditions and demonstrate its robustness by filtering noise over learning epochs, further enhancing its interpretability and stability. By integrating meta-learning principles, our method not only improves optimization stability but also enhances interpretability and generalizability, bridging the gap between NMF-based models and advanced autonomous (unsupervised) learning strategies.

Positron emission tomography (PET) is an important imaging technique to visualize a number of functions in the brain or human body. For reconstructing PET images from the sinogram data, an inverse problem has to be solved using numerical optimizations such as expectation-maximization (EM)-based methods. However, the standard EM method suffers from measurement noise added in the sinogram data. In this paper, we propose a new simultaneous PET image reconstruction and parts extraction method using constrained non-negative matrix factorization. In contrast that the many existing methods reconstruct a single PET image independently, we reconstruct the time-series of PET images simultaneously from the time-series of sinograms using non-negative matrix factorization. Furthermore, we impose the smoothness constraint for the temporal feature, and the exclusive LASSO-based sparseness constraint for the spatial feature for robust image reconstruction and physically meaningful feature extraction.

Hyperspectral unmixing (HU) is a research hotspot of hyperspectral remote sensing technology. As a classical HU method, the nonnegative matrix factorization (NMF) unmixing method can decompose an observed hyperspectral data matrix into the product of two nonnegative matrices, i.e., endmember and abundance matrices. Because the objective function of NMF is the traditional least-squares function, NMF is sensitive to noise. In order to improve the robustness of NMF, this paper proposes a maximum likelihood estimation (MLE) based NMF model (MLENMF) for unmixing of hyperspectral images (HSIs), which substitutes the least-squares objective function in traditional NMF by a robust MLE-based loss function. Experimental results on a simulated and two widely used real hyperspectral data sets demonstrate the superiority of our MLENMF over existing NMF methods.