TSSE-DMM: Topic Modeling for Short Texts Based on Topic Subdivision and Semantic Enhancement

Many applications require semantic understanding of short texts, and inferring discriminative and coherent latent topics is a critical and fundamental task in these applications. Conventional topic models largely rely on word co-occurrences to derive topics from a collection of documents. However, due to the length of each document, short texts are much more sparse in terms of word co-occurrences. Recent studies show that the Dirichlet Multinomial Mixture (DMM) model is effective for topic inference over short texts by assuming that each piece of short text is generated by a single topic. However, DMM has two main limitations. First, even though it seems reasonable to assume that each short text has only one topic because of its shortness, the definition of “shortness” is subjective and the length of the short texts is dataset dependent. That is, the single-topic assumption may be too strong for some datasets. To address this limitation, we propose to model the topic number as a Poisson distribution, allowing each short text to be associated with a small number of topics (e.g., one to three topics). This model is named PDMM. Second, DMM (and also PDMM) does not have access to background knowledge (e.g., semantic relations between words) when modeling short texts. When a human being interprets a piece of short text, the understanding is not solely based on its content words, but also their semantic relations. Recent advances in word embeddings offer effective learning of word semantic relations from a large corpus. Such auxiliary word embeddings enable us to address the second limitation. To this end, we propose to promote the semantically related words under the same topic during the sampling process, by using the generalized Pólya urn (GPU) model. Through the GPU model, background knowledge about word semantic relations learned from millions of external documents can be easily exploited to improve topic modeling for short texts. By directly extending the PDMM model with the GPU model, we propose two more effective topic models for short texts, named GPU-DMM and GPU-PDMM. Through extensive experiments on two real-world short text collections in two languages, we demonstrate that PDMM achieves better topic representations than state-of-the-art models, measured by topic coherence. The learned topic representation leads to better accuracy in a text classification task, as an indirect evaluation. Both GPU-DMM and GPU-PDMM further improve topic coherence and text classification accuracy. GPU-PDMM outperforms GPU-DMM at the price of higher computational costs.

With the emerging of massive short texts, e.g., social media posts and question titles from Q&A systems, discovering valuable information from them is increasingly significant for many real-world applications of content analysis. The family of topic modeling can effectively explore the hidden structures of documents through the assumptions of latent topics. However, due to the sparseness of short texts, the existing topic models, e.g., latent Dirichlet allocation, lose effectiveness on them. To this end, an effective solution, namely Dirichlet multinomial mixture (DMM), supposing that each short text is only associated with a single topic, indirectly enriches document-level word co-occurrences. However, DMM is sensitive to noisy words, where it often learns inaccurate topic representations at the document level. To address this problem, we extend DMM to a novel Laplacian Dirichlet Multinomial Mixture (LapDMM) topic model for short texts. The basic idea of LapDMM is to preserve local neighborhood structures of short texts, enabling to spread topical signals among neighboring documents, so as to modify the inaccurate topic representations. This is achieved by incorporating the variational manifold regularization into the variational objective of DMM, constraining the close short texts with similar variational topic representations. To find nearest neighbors of short texts, before model inference, we construct an offline document graph, where the distances of short texts can be computed by the word mover’s distance. We further develop an online version of LapDMM, namely Online LapDMM, to achieve inference speedup on massive short texts. Carrying this implications, we exploit the spirit of stochastic optimization with mini-batches and an up-to-date document graph that can efficiently find approximate nearest neighbors instead. To evaluate our models, we compare against the state-of-the-art short text topic models on several traditional tasks, i.e., topic quality, document clustering and classification. The empirical results demonstrate that our models achieve very significant performance gains over the baseline models.

Social networks such as Twitter, Facebook, and Sina microblogs have emerged as major sources for discovering and sharing the latest topics. Because social network topics change quickly, developing an effective method to model such topics is urgently needed. However, topic modeling is challenging due to the sparsity problem and the dynamic change of topics in microblog streams. In this study, we propose dynamic topic modeling via a self-aggregation method (SADTM) that can capture the time-varying aspect of topic distributions and resolve the sparsity problem. The SADTM aggregates the observable and unordered short texts as the aggregated document without leveraging an external context to overcome the sparsity problem of short text. Furthermore, we exploit word pairs instead of words for each microblog to generate more word co-occurrence patterns. The SADTM models temporal dynamics by using the topic distribution at previous time steps in microblog streams to infer the current topic from sequential data. Extensive experiments on a real-world Sina microblog dataset demonstrate that our SADTM algorithm outperforms several state-of-the-art methods in topic coherence and cluster quality. Additionally, when applied in a search scene, our SADTM significantly outperforms all baseline methods in terms of the quality of the search results.

In this article, first a comprehensive study of the impact of term weighting schemes on the topic modeling performance (i.e., LDA and DMM) on Arabic long and short texts is presented. We investigate six term weighting methods including Word count method (standard topic models), TFIDF, PMI, BDC, CLPB, and CEW. Moreover, we propose a novel combination term weighting scheme, namely, CmTLB. We utilize the mTFIDF that takes into account the missing terms and the number of the documents in which the term appears when calculating the term weight. For further robust term weight, we combine mTFIDF with two weighting methods. We evaluate CmTLB against the studied weighting schemes by the quality of the learned topics (topic visualization and topic coherence), classification, and clustering tasks. We applied weighting schemes to Latent Dirichlet allocation (LDA) and Dirichlet multinomial mixture (DMM) on eight Arabic long and short document datasets, respectively. The experiment results outline that appropriate weighting schemes can effectively improve topic modeling performance on Arabic texts. More importantly, our proposed CmTLB significantly outperforms the other weighting schemes. Secondly, we investigate whether the Arabic stemming process can improve topic modeling performance. We study the three approaches of Arabic stemming including root-based, stem-based, and statistical approaches. We also train topic models with weighting schemes on documents after applying four stemmers related to different stemming approaches. The results outline that applying the stemming process not only reduces the dimensionality of term-document matrix leading to fast estimation process, but also show enhancement of topic modeling performance both on short and long Arabic documents. Moreover, Farasa stemmer achieves the highest performance in most cases, since it prevents the ambiguity that may happen because of the blind removal of the affixes such as in root-based or stem-based stemmers.

The first new species of Leuctridae (Plecoptera) from Jiangsu Province of eastern coastal China, Rhopalopsole vespertilio sp. nov. is described and illustrated. The new species belongs to the R. magnicerca species group and is distinguished by the bat-like shaped central plate of tergum 10, by the medially narrowed subtrapezoidal transverse bars, and by the upcurved hook formed by the small epiproct. The new species is compared with similar taxa.

A wide range of web corpora are in the form of short text, such as QA queries, search queries and news titles. Entity linking for these short texts is quite important. Most of supervised approaches are not effective for short text entity linking. The training data for supervised approaches are not suitable for short text and insufficient for low-resourced languages. Previous unsupervised methods are incapable of handling the sparsity and noisy problem of short text. We try to solve the problem by mapping the sparse short text to a topic space. We notice that the concepts of entities have rich topic information and characterize entities in a very fine-grained granularity. Hence, we use the concepts of entities as topics to explicitly represent the context, which helps improve the performance of entity linking for short text. We leverage our linking approach to segment the short text semantically, and build a system for short entity text recognition and linking. Our entity linking approach exhibits the state-of-the-art performance on several datasets for the realistic short text entity linking problem.

Short text classification methods have achieved significant progress and wide application on text data such as Twitter and Weibo. However, the extremely short chinese texts like tax invoice data are different with traditional short texts in lackness of contextual semantic information, feature sparseness and extremely short length. The existing short text classification methods are difficult to achieve a satisfactory performance in these texts. To address these problems, this paper proposes a text classification method based on bidirectional semantic extension for extremely short texts like Chinese tax invoice data. More specifically, firstly, the Chinese knowledge graph is introduced for extending bidirectional semantic of texts and label data to expand the extremely short texts and ease the problem of feature sparseness; secondly, the hash vectorization is used to avoid the semantic problem caused by the lackness of contextual information. Experimental results conducted the real tax invoice dataset demonstrate the effectiveness of our proposed method.

Short text differs from traditional documents in its shortness and sparseness. Feature extension can ease the problem of high sparseness in the vector space model, but it inevitably introduces noise. To resolve this problem, this paper proposes a high-frequency feature expansion method based on a latent Dirichlet allocation (LDA) topic model. High-frequency features are extracted from each category as the feature space, using LDA to derive latent topics from the corpus, and topic words are extended to the short text. Extensive experiments are conducted on Chinese short messages and news titles. The proposed method for classifying Chinese short texts outperforms conventional classification methods.

Over the past few decades, the world has witnessed an unprecedented explosion of information. Of these, a substantial portion consists of unlabeled textual data, such as tweets, news articles, product reviews, and web snippets.
\nAs labeling is extremely expensive, time-consuming, and sometimes biased,
\nhow to effectively analyze these data becomes an imperative.
\nOwing to this, Neural Topic Models (NTMs) have emerged as a promising solution that attracts considerable research attention for their capability and interpretability.
\nThey automatically discover latent topics from unlabeled textual data through neural networks, enabling unsupervised document understanding.
\nThey have derived various downstream applications, such as content recommendation, trend analysis, and text summarization.
\nCompared to conventional topic models like LDA, NTMs offer structural flexibility and also support gradient back-propagation,
\nwhich avoids complicated model-specific derivations and well handle large-scale data.
\n
\nHowever, despite their promise, existing NTMs generally encounter several critical challenges. On the one hand, NTMs often produce low-quality topics, which are even incomparable to conventional models.
\nThese topics tend to be incoherent or repetitive, significantly diminishing their informativeness.
\nOn the other hand, NTMs struggle with low inference ability, leading to less accurate topic distributions for documents.
\nThis limitation greatly hinders document understanding and undermines the subsequent analysis, reasoning, or decision making processes.
\nDue to these challenges, existing NTMs are less useful and applicable for downstream tasks or applications.
\nAs a result, it is necessary to enhance NTMs to deliver more reliable and informative topic modeling.
\n
\nThis thesis aims to advance neural topic modeling by addressing these key challenges.
\nIn particular, we focus on four most popular scenarios: short-texts, cross-lingual, hierarchical, and basic neural topic modeling.
\nFirst, we propose a novel neural topic model tailored for short texts. 
\nThis model leverages a topic-semantic contrastive learning method to captures the similarity relations among short text samples, which works regardless of data augmentation availability.
\nThis refines short text representations, enriches learning signals, thus effectively alleviating the data sparsity issue of short texts and producing informative topics.
\n
\nSecond, we explore dynamic topic modeling.
\nWe present a neural dynamic topic model to track the evolution of dynamic topics by building the contrastive relations among them, rather than relying on the conventional Markov Chains. This model further explicitly excludes unassociated words from dynamic topics to enhance the alignment to their respective time slices.
\nThese improvements enable our model to reliably track topic evolution with high diversity.
\n
\nThird, we focus on the the affinity, rationality, and diversity of hierarchical topic modeling.
\nOur proposed neural hierarchical topic model ensures the sparsity and balance of cross-level topic dependencies using a transport plan dependency method.
\nMoreover, it distributes different semantic granularity to topics at different levels by disentangled decoding.
\nWith these, our model produces affinitive, rational, and diverse topic hierarchies.
\n
\nFourth, we tackle the topic collapsing issue and propose a basic neural topic model.
\nApart from the common reconstruction error, this model introduces a new embedding clustering regularization to force each topic embedding to be the center of a separately aggregated word embedding cluster in the semantic space.
\nThis produces topics with distinct semantics and effectively resolves the topic collapsing issue.
\n
\nFinally, we develop a comprehensive topic modeling toolkit that includes both previous conventional and cutting-edge neural topic models.
\nThis toolkit covers the complete pipelines of topic modeling, such as dataset preprocessing, model training, and evaluation.
\nThese improvements position our toolkit as a valuable resource to accelerate the research and applications of topic models.
\n
\nIn conclusion, this thesis makes significant contributions towards advancing neural topic modeling through multiple models, various scenarios, and a rigorous and comprehensive benchmark toolkit.
\nThese contributions pave the way for the utilization of neural topic modeling in diverse real-world applications.

In this paper we develop an agent-based model to explore a feasible way of simultaneously providing sufficient food and bioenergy feedstocks in China. Concerns over the competition for agricultural land resources between food and bioenergy supply hinder the further development of bioenergy, especially in China, the country that needs to feed the world’s largest population. Prior research has suggested the introduction of energy crops and reviewed the resulting agricultural land use change in China. However, there is a lack of quantitative studies which estimate the value, contribution, and impact of bioenergy for specific conditions at the county level and provide adequate information to guide local practices. To fill this gap, we choose the Jiangsu Province in China as a case study, build up a spatial-agent dynamic model of agricultural land use, and perform a sensitivity analysis for important parameters. The simulation results show that straw from conventional crops generally dominates Jiangsu’s biomass supply with a contribution above 85%. The sensitivity analyses reveal severe consequences of bioenergy targets for local land use. For Jiangsu Province, reclaimed mudflats, an alternative to arable lands for energy crop plantation, help to secure the local biomass supply and to alleviate the land use conflict between food and biomass production.

The DEA-Malmquist model was used to measure the ecotourism efficiency of 13 cities in Jiangsu Province from 2010 to 2020, and the spatial and temporal differences were further analysed. The results show that the tourism technical efficiency in Jiangsu Province in China is generally at a high level, and the tourism scale efficiency restricts the improvement of tourism technical efficiency. The unbalanced of the technical progress in tourism between the prefecture-level cities is obvious. Overall, the change of technical progress in tourism has the strongest impact on the total factor productivity index. Therefore, strengthening the inter-regional cooperation of tourism development, training of tourism professionals, improving the structure of tourism industry, optimizing the allocation of tourism production factors, and expanding the scale and agglomeration effect of tourism can improve the efficiency of ecotourism.

BackgroundTopic models are a class of unsupervised machine learning models, which facilitate summarization, browsing and retrieval from large unstructured document collections. This study reviews several methods for assessing the quality of unsupervised topic models estimated using non-negative matrix factorization. Techniques for topic model validation have been developed across disparate fields. We synthesize this literature, discuss the advantages and disadvantages of different techniques for topic model validation, and illustrate their usefulness for guiding model selection on a large clinical text corpus.Design, setting and dataUsing a retrospective cohort design, we curated a text corpus containing 382,666 clinical notes collected between 01/01/2017 through 12/31/2020 from primary care electronic medical records in Toronto Canada.MethodsSeveral topic model quality metrics have been proposed to assess different aspects of model fit. We explored the following metrics: reconstruction error, topic coherence, rank biased overlap, Kendall’s weighted tau, partition coefficient, partition entropy and the Xie-Beni statistic. Depending on context, cross-validation and/or bootstrap stability analysis were used to estimate these metrics on our corpus.ResultsCross-validated reconstruction error favored large topic models (K ≥ 100 topics) on our corpus. Stability analysis using topic coherence and the Xie-Beni statistic also favored large models (K = 100 topics). Rank biased overlap and Kendall’s weighted tau favored small models (K = 5 topics). Few model evaluation metrics suggested mid-sized topic models (25 ≤ K ≤ 75) as being optimal. However, human judgement suggested that mid-sized topic models produced expressive low-dimensional summarizations of the corpus.ConclusionsTopic model quality indices are transparent quantitative tools for guiding model selection and evaluation. Our empirical illustration demonstrated that different topic model quality indices favor models of different complexity; and may not select models aligning with human judgment. This suggests that different metrics capture different aspects of model goodness of fit. A combination of topic model quality indices, coupled with human validation, may be useful in appraising unsupervised topic models.

The boost of short texts is increasing demands for short text classification. Feature extraction based on topic models, especially those utilizing label information, such as Partially Labeled Dirichlet Allocation (PLDA), has been widely employed in normal text classification. However, the application of topic models in short text classification is still an open challenge due to the shortness and sparse nature of short texts. In this paper, we propose a topic model called Short Text PLDA (ST-PLDA) based on PLDA for short text classification. In ST-PLDA, each short text is restricted to one label and assumed to be generated from a single topic associated with its label and a background topic shared by all texts. Thus, ST-PLDA is expected to discover more compact and discriminative topic representations. Extensive experiments are conducted on real-world short text collections and the promising results demonstrate the superiority of ST-PLDA compared with other state-of-the-art methods.

Topic modeling extracts useful potential topics that reflect market information from massive financial news and is widely used in data mining and economic research. Traditional topic modeling approaches such as Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) lack semantic information, and short texts have feature sparse problems. We develop a topic clustering model based on BERT-LDA joint embedding that takes both contextual semantics and thematic narrative into account. We cluster document embeddings with the HDBSCAN algorithm and utilize a class-based TF-IDF (c-TF-IDF) method to create topic representations. Empirical results show that the BERT-LDA model is competitive compared with traditional and single topic models. It generates coherent topic words that are dissimilar to each other.

Local and global topics in text modeling of web pages nested in web sites