Quality indices for topic model selection and evaluation: a literature review and case study

Background/Objectives: Unsupervised topic models are often used to facilitate improved understanding of large unstructured clinical text datasets. In this study we investigated how ICD-9 diagnostic codes, collected alongside clinical text data, could be used to establish concurrent-, convergent- and discriminant-validity of learned topic models. Design/Setting: Retrospective open cohort design. Data were collected from primary care clinics located in Toronto, Canada between 01/01/2017 through 12/31/2020. Methods: We fit a non-negative matrix factorization topic model, with K = 50 latent topics/themes, to our input document term matrix (DTM). We estimated the magnitude of association between each Boolean-valued ICD-9 diagnostic code and each continuous latent topical vector. We identified ICD-9 diagnostic codes most strongly associated with each latent topical vector; and qualitatively interpreted how these codes could be used for external validation of the learned topic model. Results: The DTM consisted of 382,666 documents and 2210 words/tokens. We correlated concurrently assigned ICD-9 diagnostic codes with learned topical vectors, and observed semantic agreement for a subset of latent constructs (e.g. conditions of the breast, disorders of the female genital tract, respiratory disease, viral infection, eye/ear/nose/throat conditions, conditions of the urinary system, and dermatological conditions, etc.). Conclusions: When fitting topic models to clinical text corpora, researchers can leverage contemporaneously collected electronic medical record data to investigate the external validity of fitted latent variable models.

Topic models are often evaluated with measures such as perplexity and topic coherence.However, these methods fall short in determining the comprehensiveness of identified topics.This research introduces a complementary approach to evaluating unsupervised topic models using a labeled dataset.By training hierarchical topic models and utilizing known labels for evaluation, we found a high accuracy of 70% for expected topics.Despite having 90 labels in the dataset, even those representing only 1% of the data achieved an average accuracy of 37.9%, illustrating hierarchical topic models' effectiveness on smaller subsets.Additionally, we confirmed that this new evaluation method helps assess the topic tree quality, demonstrating that hierarchical topic models generate coherent taxonomies.Lastly, we established that coherence measures alone are insufficient for a holistic topic model evaluation.

Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improve the quality of these token-level topic assignments, have been evaluated only with respect to global metrics. We propose a task designed to elicit human judgments of token-level topic assignments. We use a variety of topic model types and parameters and discover that global metrics agree poorly with human assignments. Since human evaluation is expensive we propose a variety of automated metrics to evaluate topic models at a local level. Finally, we correlate our proposed metrics with human judgments from the task on several datasets. We show that an evaluation based on the percent of topic switches correlates most strongly with human judgment of local topic quality. We suggest that this new metric, which we call consistency, be adopted alongside global metrics such as topic coherence when evaluating new topic models.

Topic modeling extracts useful potential topics that reflect market information from massive financial news and is widely used in data mining and economic research. Traditional topic modeling approaches such as Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) lack semantic information, and short texts have feature sparse problems. We develop a topic clustering model based on BERT-LDA joint embedding that takes both contextual semantics and thematic narrative into account. We cluster document embeddings with the HDBSCAN algorithm and utilize a class-based TF-IDF (c-TF-IDF) method to create topic representations. Empirical results show that the BERT-LDA model is competitive compared with traditional and single topic models. It generates coherent topic words that are dissimilar to each other.

In an effort to develop topic modeling methods that can be quickly applied to large data sets, we revisit the problem of maximum-likelihood estimation in topic models. It is known, at least informally, that maximum-likelihood estimation in topic models is closely related to non-negative matrix factorization (NMF). Yet, to our knowledge, this relationship has not been exploited previously to fit topic models. We show that recent advances in NMF optimization methods can be leveraged to fit topic models very efficiently, often resulting in much better fits and in less time than existing algorithms for topic models. We also formally make the connection between the NMF optimization problem and maximum-likelihood estimation for the topic model, and using this result we show that the expectation maximization (EM) algorithm for the topic model is essentially the same as the classic multiplicative updates for NMF. Our methods are implemented in the R package "fastTopics".

Recent years have seen the rise of subject-themed digital libraries, such as the NSDL pathways and the Digital Library for Earth System Education (DLESE). These libraries often need to manually verify that contributed resources cover topics that fit within the theme of the library. We show that such scope judgments can be automated using a combination of text classification techniques and topic modeling. Our models address two significant challenges in making scope judgments: only a small number of out-of-scope resources are typically available, and the topic distinctions required for digital libraries are much more subtle than classic text classification problems. To meet these challenges, our models combine support vector machine learners optimized to different performance metrics and semantic topics induced by unsupervised statistical topic models. Our best model is able to distinguish resources that belong in DLESE from resources that don't with an accuracy of around 70%. We see these models as the first steps towards increasing the scalability of digital libraries and dramatically reducing the workload required to maintain them.

An analysis of the coherence of descriptors in topic modeling

With technologies that afford much larger-scale data collection than previously imagined, new ways of processing and interpreting qualitative textual data are required. HCI researchers use a range of methods for interpreting the 'full range of human experience' from qualitative data, however, such approaches are not always scalable. Feminist geography seeks to explore how diverse and varied accounts of place can be understood and represented, whilst avoiding reductive classification systems. In this paper, we assess the extent to which unsupervised topic models can support such a research agenda. Drawing on literature from Feminist and Critical GIS, we present a case study analysis of a Volunteered Geographic Information dataset of reviews about breastfeeding in public spaces. We demonstrate that topic modelling can offer novel insights and nuanced interpretations of complex concepts such as privacy and be integrated into a critically reflexive feminist data analysis approach that captures and represents diverse experiences of place.

Purpose: This study aims to propose an unsupervised topic modeling algorithm utilizing contrastive learning to overcome the limitations of existing methods.Methods: The proposed algorithm incorporates contrastive learning into topic modeling and employs an end-to-end learning process. This approach embeds documents, topics, and words into a unified space, identifying their relationships and providing more intuitive topic modeling results. The algorithm was applied to scientific and technical literature data and its performance was evaluated against existing topic modeling methods.Results: The proposed algorithm demonstrated comparable or superior quantitative metrics in terms of consistency and diversity when compared to existing topic modeling approaches on scientific and technical literature datasets. These results indicate that the embeddings learned through contrastive learning effectively capture the relationships between documents and topics.Conclusion: The unsupervised topic modeling algorithm leveraging contrastive learning provides more intuitive results compared to traditional topic modeling approaches and proves to be a valuable tool for large-scale text data analysis. Future research can explore its application in various domains and evaluate the algorithm using additional quantitative and qualitative metrics to further expand its applicability.

©2016 Association for Computational Linguistics. When evaluating the quality of topics generated by a topic model, the convention is to score topic coherence - either manually or automatically - using the top-N topic words. This hyper-parameter N, or the cardinality of the topic, is often overlooked and selected arbitrarily. In this paper, we investigate the impact of this cardinality hyper-parameter on topic coherence evaluation. For two automatic topic coherence methodologies, we observe that the correlation with human ratings decreases systematically as the cardinality increases. More interestingly, we find that performance can be improved if the system scores and human ratings are aggregated over several topic cardinalities before computing the correlation. In contrast to the standard practice of using a fixed value of N (e.g. N = 5 or N = 10), our results suggest that calculating topic coherence over several different cardinalities and averaging results in a substantially more stable and robust evaluation. We release the code and the datasets used in this research, for reproducibility.1

Topic modeling continues to grow as a popular technique for finding hidden patterns, as well as grouping collections of new types of text and non-text data. Recent years have witnessed a growing body of work in developing metrics and techniques for evaluating the quality of topic models and the topics they generate. This is particularly true for text data where significant attention has been given to the semantic interpretability of topics using measures such as coherence. It has been shown however that topic assessments based on coherence metrics do not always align well with human judgment. Other efforts have examined the utility of information-theoretic distance metrics for evaluating topic quality in connection with semantic interpretability. Although there has been progress in evaluating interpretability of topics, the existing intrinsic evaluation metrics do not address some of the other aspects of concern in topic modeling such as: the number of topics to select, the ability to align topics from different models, and assessing the quality of training data. Here we propose an alternative metric for characterizing topic quality that addresses all three aforementioned issues. Our approach is based on clustering topics, and using the silhouette measure, a popular clustering index, for characterizing the quality of topics. We illustrate the utility of this approach in addressing the other topic modeling concerns noted above. Since this metric is not focused on interpretability, we believe it can be applied more broadly to text as well as non-text data. In this paper however we focus on the application of this metric to archival and non-archival text data.

Latent Dirichlet Allocation (LDA) is a topic model which has been applied to various fields. It has been also applied to user profiling or event summarization on Twitter. In the application of LDA to tweet collection, it generally treats aggregated all tweets of a user as a single document. On the other hand, Twitter-LDA which assumes a single tweet consists of a single topic has been proposed and showed that it is superior to the former way in topic semantic coherence. However, Twitter-LDA has a problem that it is not capable of online inference. In this paper, we extend Twitter-LDA in the following two points. First, we model the generation process of tweets more accurately by estimating the ratio between topic words and general words for each user. Second, we enable it to estimate dynamics of user interests and topic trends in online based on Topic Tracking Model (TTM) which models consumer purchase behaviors.

Nan-negative matrix factorization (NAfP) topic modeling has recently been introduced for the categorization and analysis of crime report text. Topic modeling in Otis context allows for more nuanced categories of crime compared to official UCR categorizations. In this paper we suggest two metrics for the evaluation of crime topic models: coherence and spatial concentration. The importance of space comes into play through Weisburd's law of crime concentration, that states a large percentage of crime occurs in a small area of a city. We investigate the extent to which topic models that improve coherence lead to higher levels of crime concentration. Through analyzing a dataset of crime reports from Los Angeles, CA, we find that Latent Dirichlst Allocation (IDA) generates crime topics with both higher coherence and crime concentration. While NMF improves the coherence compared to UCR categorization, die spatial concentration is not as high. These foldings have important implications for hotspot policing.

In this paper, we study the digital marketing where marketing officers (MOs) have to commit to creating brand new promotion ads/contents based on understandings of users' needs or preferences. Users' behaviors are typically high dimensional and hard to understand. Therefore, dimension reduction of users' behaviors from high dimensions and explainability are important to help MOs launch operation-friendly marketings. As such, it is natural to exploit topic models to help MOs understand users' intents from users' behaviors (e.g., user-item visits) in case we treat each user as a document and users' behaviors of visiting an item as a word. However, users of low activities and items followed by power law distributions are common in user-item visit data, which pose significant challenges to traditional topic models. We present a social and semantic enhanced topic model (S²TM) for users' intent mining. We optimize the user-intent estimates based on a graph neural network atop of a social network, and optimize the intent-item estimates based on a skip-gram word embedding approach by linking the semantics of items to pre-trained word embeddings. We propose an efficient stochastic vari-ational inference algorithm for the inference of latent variables and learning of parameters. Extensive experiments on real-world data show the effectivenesses of S²TM in terms of perplexities, topic coherence and semantic coherence compared with state-of-the-art topic models. We further show how MOs interact with our operation-friendly intent mining system, and results on real-world marketing campaigns in terms of click-through rate at Alipay.

In this article, first a comprehensive study of the impact of term weighting schemes on the topic modeling performance (i.e., LDA and DMM) on Arabic long and short texts is presented. We investigate six term weighting methods including Word count method (standard topic models), TFIDF, PMI, BDC, CLPB, and CEW. Moreover, we propose a novel combination term weighting scheme, namely, CmTLB. We utilize the mTFIDF that takes into account the missing terms and the number of the documents in which the term appears when calculating the term weight. For further robust term weight, we combine mTFIDF with two weighting methods. We evaluate CmTLB against the studied weighting schemes by the quality of the learned topics (topic visualization and topic coherence), classification, and clustering tasks. We applied weighting schemes to Latent Dirichlet allocation (LDA) and Dirichlet multinomial mixture (DMM) on eight Arabic long and short document datasets, respectively. The experiment results outline that appropriate weighting schemes can effectively improve topic modeling performance on Arabic texts. More importantly, our proposed CmTLB significantly outperforms the other weighting schemes. Secondly, we investigate whether the Arabic stemming process can improve topic modeling performance. We study the three approaches of Arabic stemming including root-based, stem-based, and statistical approaches. We also train topic models with weighting schemes on documents after applying four stemmers related to different stemming approaches. The results outline that applying the stemming process not only reduces the dimensionality of term-document matrix leading to fast estimation process, but also show enhancement of topic modeling performance both on short and long Arabic documents. Moreover, Farasa stemmer achieves the highest performance in most cases, since it prevents the ambiguity that may happen because of the blind removal of the affixes such as in root-based or stem-based stemmers.