Topic extraction from extremely short texts with variational manifold regularization

Many applications require semantic understanding of short texts, and inferring discriminative and coherent latent topics is a critical and fundamental task in these applications. Conventional topic models largely rely on word co-occurrences to derive topics from a collection of documents. However, due to the length of each document, short texts are much more sparse in terms of word co-occurrences. Recent studies show that the Dirichlet Multinomial Mixture (DMM) model is effective for topic inference over short texts by assuming that each piece of short text is generated by a single topic. However, DMM has two main limitations. First, even though it seems reasonable to assume that each short text has only one topic because of its shortness, the definition of “shortness” is subjective and the length of the short texts is dataset dependent. That is, the single-topic assumption may be too strong for some datasets. To address this limitation, we propose to model the topic number as a Poisson distribution, allowing each short text to be associated with a small number of topics (e.g., one to three topics). This model is named PDMM. Second, DMM (and also PDMM) does not have access to background knowledge (e.g., semantic relations between words) when modeling short texts. When a human being interprets a piece of short text, the understanding is not solely based on its content words, but also their semantic relations. Recent advances in word embeddings offer effective learning of word semantic relations from a large corpus. Such auxiliary word embeddings enable us to address the second limitation. To this end, we propose to promote the semantically related words under the same topic during the sampling process, by using the generalized Pólya urn (GPU) model. Through the GPU model, background knowledge about word semantic relations learned from millions of external documents can be easily exploited to improve topic modeling for short texts. By directly extending the PDMM model with the GPU model, we propose two more effective topic models for short texts, named GPU-DMM and GPU-PDMM. Through extensive experiments on two real-world short text collections in two languages, we demonstrate that PDMM achieves better topic representations than state-of-the-art models, measured by topic coherence. The learned topic representation leads to better accuracy in a text classification task, as an indirect evaluation. Both GPU-DMM and GPU-PDMM further improve topic coherence and text classification accuracy. GPU-PDMM outperforms GPU-DMM at the price of higher computational costs.

In this article, first a comprehensive study of the impact of term weighting schemes on the topic modeling performance (i.e., LDA and DMM) on Arabic long and short texts is presented. We investigate six term weighting methods including Word count method (standard topic models), TFIDF, PMI, BDC, CLPB, and CEW. Moreover, we propose a novel combination term weighting scheme, namely, CmTLB. We utilize the mTFIDF that takes into account the missing terms and the number of the documents in which the term appears when calculating the term weight. For further robust term weight, we combine mTFIDF with two weighting methods. We evaluate CmTLB against the studied weighting schemes by the quality of the learned topics (topic visualization and topic coherence), classification, and clustering tasks. We applied weighting schemes to Latent Dirichlet allocation (LDA) and Dirichlet multinomial mixture (DMM) on eight Arabic long and short document datasets, respectively. The experiment results outline that appropriate weighting schemes can effectively improve topic modeling performance on Arabic texts. More importantly, our proposed CmTLB significantly outperforms the other weighting schemes. Secondly, we investigate whether the Arabic stemming process can improve topic modeling performance. We study the three approaches of Arabic stemming including root-based, stem-based, and statistical approaches. We also train topic models with weighting schemes on documents after applying four stemmers related to different stemming approaches. The results outline that applying the stemming process not only reduces the dimensionality of term-document matrix leading to fast estimation process, but also show enhancement of topic modeling performance both on short and long Arabic documents. Moreover, Farasa stemmer achieves the highest performance in most cases, since it prevents the ambiguity that may happen because of the blind removal of the affixes such as in root-based or stem-based stemmers.

With the rapid development of social media and e-commerce on the Internet, many short texts such as instant messages, tweets and product comments have become an important form of Internet information. Compared with regular texts, short texts suffer severe data sparsity and informal format. Because of that, mining latent semantics such as topics in short texts is a critical challenge. There are many limitations directly applying common topic models like Latent Dirichlet Allocation (LDA) to short texts. In result of that, many short text topic models were put forward. The Word Network Topic Model (WNTM) is the typical one among them, which transfers sparse document to word space into dense word to word space and then learn topics from it. However, WNTM has its own limitation. The word co-occurrence matrix of the original document is used to construct the word network. This method is simple and intuitive, which can't express the deep meaning between words and words. To solve this problem, we proposed W2V-WNTM, which uses Word2Vec model instead of word co–occurrence to construct word network. Experiment shows it can describe relationship between words better.

Short texts have been prevalent in Web sites and the emerging social media for several years, which makes it a critical task to identify intelligible topics from online data sources. However, the existing topic models over short texts cannot analyze the internal components of the learned topics, which is significant for improving the coherence and interpretability of topics. In this paper, we propose a novel topic model for short texts, named TSSE-DMM, for improving the coherence and interpretability of topics by the topic subdivision and alleviating the problem of text sparsity by the semantic enhancement strategy. Firstly, we subdivide each topic into 4 detailed aspects, namely the location aspect, the people & organization aspect, the core word aspect, and the background word aspect, to obtain the different and interpretable components of topics. Then, we combine the Generalized Polya Urn model and the joint word embedding to solve the problem of data sparsity. The extensive experimental results carried on three real-world text collections in two languages show that our model achieves better topic representations than the baseline methods. Moreover, our method has been adopted by the public service hotline platform of Jiangsu province in China.

Recently, research on short text topic models has addressed the challenges of social media datasets. These models are typically evaluated using automated measures. However, recent work suggests that these evaluation measures do not inform whether the topics produced can yield meaningful insights for those examining social media data. Efforts to address this issue, including gauging the alignment between automated and human evaluation tasks, are hampered by a lack of knowledge about how researchers use topic models. Further problems could arise if researchers do not construct topic models optimally or use them in a way that exceeds the models’ limitations. These scenarios threaten the validity of topic model development and the insights produced by researchers employing topic modelling as a methodology. However, there is currently a lack of information about how and why topic models are used in applied research. As such, we performed a systematic literature review of 189 articles where topic modelling was used for social media analysis to understand how and why topic models are used for social media analysis. Our results suggest that the development of topic models is not aligned with the needs of those who use them for social media analysis. We have found that researchers use topic models sub-optimally. There is a lack of methodological support for researchers to build and interpret topics. We offer a set of recommendations for topic model researchers to address these problems and bridge the gap between development and applied research on short text topic models.

Topic modeling extracts useful potential topics that reflect market information from massive financial news and is widely used in data mining and economic research. Traditional topic modeling approaches such as Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) lack semantic information, and short texts have feature sparse problems. We develop a topic clustering model based on BERT-LDA joint embedding that takes both contextual semantics and thematic narrative into account. We cluster document embeddings with the HDBSCAN algorithm and utilize a class-based TF-IDF (c-TF-IDF) method to create topic representations. Empirical results show that the BERT-LDA model is competitive compared with traditional and single topic models. It generates coherent topic words that are dissimilar to each other.

A short text topic modeling method based on integrating Gaussian and Logistic coding networks with pre-trained word embeddings

BackgroundCaring for family caregivers of dementia patients has grown to an important topic. Social media platforms, like Twitter, provide great resources for studying the needs of caregivers. It would be beneficial to understand the caregivers’ interested or concerned topics from their tweets. Meanwhile, topic modeling has become an efficient NLP tool for analyzing semantic themes within texts. Thus, in this study, we perform a novel topic model leveraging weighting strategies to analyze dementia‐caregiver‐related twitter data.MethodThe information about the twitter data collection is available at: https://aging.jmir.org/2022/3/e39547. In our analysis, we first performed quality control, as shown in Figure 1. Then, we deployed two word‐frequency weighting strategies, Log weighting and Balanced Distributional Concentration (BDC) weighting, on the Dirichlet Multinomial Mixture (DMM) model to obtain 4 DMM variants and used them to mine semantic topics within the tweets. We compared the performance of DMM variants, LDA variants, and BERTopic variants. Topic numbers of 5, 10, 15, 20 were picked to achieve comprehensive comparison. The coherence score C_V was applied to evaluate the model performance. We then leveraged the GPT‐4 to summarize the bag of words in each topic from the best model to generate interpretable themes.Result224,862 tweets passed the quality control and were analyzed in our study. As shown in Figure 2, the performance evaluation demonstrated that the Log‐BDC DMM has the best general performance among traditional and BERT‐based models, with an averaged coherence score of 0.5648. The bags of words from the Log‐BDC DMM with 10 topics were interpreted using GPT‐4 due to the highest coherence score of 0.6054. The interpretation yielded 10 semantic themes, as shown in Figure 3.ConclusionOur investigation of the twitter data shows that the DMM model with Log and BDC strategies has promising power for mining meaningful semantic topics from short and noisy tweets. The model outperforms LDA models and the BERTopic models while maintaining high model interpretability. This initial study on twitter data provides interesting findings that can provide better care to family caregivers and demonstrates the promise and interpretability of traditional topic modeling methods.

In the past few years, we have witnessed a rapid development of online social media, from which we can access various short texts. Understanding the topic patterns of these short text is significant. Traditional topic models, like LDA, are not suitable when applied to short text topic analysis due to data sparsity. A lot of efforts have been made to solve this problem. However, there is still significant space to improve the effectiveness of these short text specific methods. In this paper, we proposed a novel word co-occurrence network based method, referred to as biterm pseudo document topic model (BPDTM), which extended the previous biterm topic model(BTM) for short text. We utilized the word co-occurrence network to construct biterm pseudo documents. The proposed model is promising since it represents words with their semantic adjacent biterms and is able to model the corpus-level semantic relation between two words. Besides, BPDTM naturally lengthens the documents, which alleviate the influence for performance exerted by data sparsity. Experiments demonstrated that our model outperformed two baselines, i.e. LDA and BTM, which proved its effectiveness on short text topic modeling task.

In this paper, we propose a method for short text categorization using topic model and integrated classifier. To enrich the representation of short text, the Latent Dirichlet Allocation (LDA) model is used to extract latent topic information. While for classification, we combine two classifiers for achieving high reliability. Particularly, we train LDA models with variable number of topics using the Wikipedia corpus as external knowledge base, and extend labeled Web snippets by potential topics extracted by LDA. Then, the enriched representation of snippets are used to learn Maximum Entropy (MaxEnt) and support vector machine (SVM) classifiers separately. Finally, viewing that the most possible predicted result will appear in the top two candidates selected by MaxEnt classifier, we develop a novel scheme that if the gap between these candidates is large enough, the predicted result is considered to be reliable; otherwise, the SVM classifier will be integrated with MaxEnt classifier to make a comprehensive prediction. Experimental results show that our framework is effective and can outperform the state-of-the-art techniques.

Short text differs from traditional documents in its shortness and sparseness. Feature extension can ease the problem of high sparseness in the vector space model, but it inevitably introduces noise. To resolve this problem, this paper proposes a high-frequency feature expansion method based on a latent Dirichlet allocation (LDA) topic model. High-frequency features are extracted from each category as the feature space, using LDA to derive latent topics from the corpus, and topic words are extended to the short text. Extensive experiments are conducted on Chinese short messages and news titles. The proposed method for classifying Chinese short texts outperforms conventional classification methods.

Over the past few decades, the world has witnessed an unprecedented explosion of information. Of these, a substantial portion consists of unlabeled textual data, such as tweets, news articles, product reviews, and web snippets.
\nAs labeling is extremely expensive, time-consuming, and sometimes biased,
\nhow to effectively analyze these data becomes an imperative.
\nOwing to this, Neural Topic Models (NTMs) have emerged as a promising solution that attracts considerable research attention for their capability and interpretability.
\nThey automatically discover latent topics from unlabeled textual data through neural networks, enabling unsupervised document understanding.
\nThey have derived various downstream applications, such as content recommendation, trend analysis, and text summarization.
\nCompared to conventional topic models like LDA, NTMs offer structural flexibility and also support gradient back-propagation,
\nwhich avoids complicated model-specific derivations and well handle large-scale data.
\n
\nHowever, despite their promise, existing NTMs generally encounter several critical challenges. On the one hand, NTMs often produce low-quality topics, which are even incomparable to conventional models.
\nThese topics tend to be incoherent or repetitive, significantly diminishing their informativeness.
\nOn the other hand, NTMs struggle with low inference ability, leading to less accurate topic distributions for documents.
\nThis limitation greatly hinders document understanding and undermines the subsequent analysis, reasoning, or decision making processes.
\nDue to these challenges, existing NTMs are less useful and applicable for downstream tasks or applications.
\nAs a result, it is necessary to enhance NTMs to deliver more reliable and informative topic modeling.
\n
\nThis thesis aims to advance neural topic modeling by addressing these key challenges.
\nIn particular, we focus on four most popular scenarios: short-texts, cross-lingual, hierarchical, and basic neural topic modeling.
\nFirst, we propose a novel neural topic model tailored for short texts. 
\nThis model leverages a topic-semantic contrastive learning method to captures the similarity relations among short text samples, which works regardless of data augmentation availability.
\nThis refines short text representations, enriches learning signals, thus effectively alleviating the data sparsity issue of short texts and producing informative topics.
\n
\nSecond, we explore dynamic topic modeling.
\nWe present a neural dynamic topic model to track the evolution of dynamic topics by building the contrastive relations among them, rather than relying on the conventional Markov Chains. This model further explicitly excludes unassociated words from dynamic topics to enhance the alignment to their respective time slices.
\nThese improvements enable our model to reliably track topic evolution with high diversity.
\n
\nThird, we focus on the the affinity, rationality, and diversity of hierarchical topic modeling.
\nOur proposed neural hierarchical topic model ensures the sparsity and balance of cross-level topic dependencies using a transport plan dependency method.
\nMoreover, it distributes different semantic granularity to topics at different levels by disentangled decoding.
\nWith these, our model produces affinitive, rational, and diverse topic hierarchies.
\n
\nFourth, we tackle the topic collapsing issue and propose a basic neural topic model.
\nApart from the common reconstruction error, this model introduces a new embedding clustering regularization to force each topic embedding to be the center of a separately aggregated word embedding cluster in the semantic space.
\nThis produces topics with distinct semantics and effectively resolves the topic collapsing issue.
\n
\nFinally, we develop a comprehensive topic modeling toolkit that includes both previous conventional and cutting-edge neural topic models.
\nThis toolkit covers the complete pipelines of topic modeling, such as dataset preprocessing, model training, and evaluation.
\nThese improvements position our toolkit as a valuable resource to accelerate the research and applications of topic models.
\n
\nIn conclusion, this thesis makes significant contributions towards advancing neural topic modeling through multiple models, various scenarios, and a rigorous and comprehensive benchmark toolkit.
\nThese contributions pave the way for the utilization of neural topic modeling in diverse real-world applications.

<span lang="EN-US">Short text topic modeling remains challenging due to data sparsity, limited word co-occurrences, and unstable clustering results, particularly for Indonesian texts. This study proposes an improved BERTopic framework that integrates IndoBERT embeddings, best match 25 (BM25)-based topic representation, and balanced iterative reducing and clustering using hierarchies (BIRCH) clustering to address these issues. IndoBERT generates contextual embeddings adapted to Indonesian linguistic features, and BM25 weighting improves keyword relevance by considering document length and term saturation. BIRCH clustering minimizes outliers by assigning most documents to valid clusters, which enhances data utilization and topic stability. Experiments on Indonesian datasets from X (formerly Twitter), Google Reviews, and YouTube demonstrate that the proposed approach consistently achieves higher topic coherence. The proposed method yields stable topic diversity values between 0.91 and 0.94, maintains embedding density from 0.60 to 0.66, and achieves intra-topic similarity between 0.39 and 0.41 across increasing dataset sizes. The proposed framework successfully reduces outlier proportions to 1-5%, which significantly outperforms standard BERTopic and K-Means. Furthermore, the model maintains stable topic counts as the data volume grows, confirming robustness and scalability for sparse short text modeling. Overall, integrating IndoBERT, BM25, and BIRCH provides a more coherent, stable, and effective solution for Indonesian short text topic modeling.</span>

The boost of short texts is increasing demands for short text classification. Feature extraction based on topic models, especially those utilizing label information, such as Partially Labeled Dirichlet Allocation (PLDA), has been widely employed in normal text classification. However, the application of topic models in short text classification is still an open challenge due to the shortness and sparse nature of short texts. In this paper, we propose a topic model called Short Text PLDA (ST-PLDA) based on PLDA for short text classification. In ST-PLDA, each short text is restricted to one label and assumed to be generated from a single topic associated with its label and a background topic shared by all texts. Thus, ST-PLDA is expected to discover more compact and discriminative topic representations. Extensive experiments are conducted on real-world short text collections and the promising results demonstrate the superiority of ST-PLDA compared with other state-of-the-art methods.

Service search in IoT's large-scale, heterogeneous and multi-domain services space is a challenging task. It can take the time that may not be acceptable for many IoT applications and requires resources that may not be available in many IoT devices. A categorisation of these services into their application domains can reduce the search space and offer an efficient and scalable service search. Recently, in many fields, such as short text messages and categorisation of IoT service specifications, generative probabilistic models, like Topic modelling, are being used. Generally, IoT service descriptions are short and sparse. Existing work on IoT services categorisation is based on Latent Dirichlet Allocation (LDA), but it does not perform well in short and sparse texts. Also, IoT services categorisation has few specific issues, which are not well addressed by existing short texts-specific topic modelling approaches. In this paper, we identify these issues and quantitatively and qualitatively evaluate how well a set of selected short texts-specific topic modelling approaches perform as IoT service categorisers against these issues. The results show that these approaches do not perform well in a corpus of noisy APIs descriptions and heterogeneous service descriptions. Also, they do not support domain identification of services, which is essential in domain-based service search. We conclude that integrating an appropriate and comprehensive knowledge base (i.e., domain ontology) could minimise noise and address IoT's APIs and service descriptions' heterogeneity. More importantly, it can identify the domains of those APIs and services.