A Topic Discovery Model Based on Content and Emotion in Microblogs

Traditional methods treat the search problem as a process of selecting and ranking sequential documents. The methods have been proved effective and are widely used in the web search domain. However, due to the complexity and particularity of microblog text contents, the classical methods are rarely used microblog searches for specific topics. Focusing on the issue of searching for specific topics in microblog content, we present a microblog search method for security topics based on deep reinforcement learning by modeling the microblog search for specific topics as a continuous-state Markov decision process. We also design a novel deep Q network to evaluate the relevance of microblog content based on the target topic. We adopt reinforcement learning to solve the microblog search problem using an intelligent strategy and evaluate content relevance through deep learning. Experiments conducted on a real-world dataset show that our approach outperforms the selected baseline methods.

With the emerging of massive short texts, e.g., social media posts and question titles from Q&A systems, discovering valuable information from them is increasingly significant for many real-world applications of content analysis. The family of topic modeling can effectively explore the hidden structures of documents through the assumptions of latent topics. However, due to the sparseness of short texts, the existing topic models, e.g., latent Dirichlet allocation, lose effectiveness on them. To this end, an effective solution, namely Dirichlet multinomial mixture (DMM), supposing that each short text is only associated with a single topic, indirectly enriches document-level word co-occurrences. However, DMM is sensitive to noisy words, where it often learns inaccurate topic representations at the document level. To address this problem, we extend DMM to a novel Laplacian Dirichlet Multinomial Mixture (LapDMM) topic model for short texts. The basic idea of LapDMM is to preserve local neighborhood structures of short texts, enabling to spread topical signals among neighboring documents, so as to modify the inaccurate topic representations. This is achieved by incorporating the variational manifold regularization into the variational objective of DMM, constraining the close short texts with similar variational topic representations. To find nearest neighbors of short texts, before model inference, we construct an offline document graph, where the distances of short texts can be computed by the word mover’s distance. We further develop an online version of LapDMM, namely Online LapDMM, to achieve inference speedup on massive short texts. Carrying this implications, we exploit the spirit of stochastic optimization with mini-batches and an up-to-date document graph that can efficiently find approximate nearest neighbors instead. To evaluate our models, we compare against the state-of-the-art short text topic models on several traditional tasks, i.e., topic quality, document clustering and classification. The empirical results demonstrate that our models achieve very significant performance gains over the baseline models.

Context reinforced neural topic modeling over short texts

Many applications require semantic understanding of short texts, and inferring discriminative and coherent latent topics is a critical and fundamental task in these applications. Conventional topic models largely rely on word co-occurrences to derive topics from a collection of documents. However, due to the length of each document, short texts are much more sparse in terms of word co-occurrences. Recent studies show that the Dirichlet Multinomial Mixture (DMM) model is effective for topic inference over short texts by assuming that each piece of short text is generated by a single topic. However, DMM has two main limitations. First, even though it seems reasonable to assume that each short text has only one topic because of its shortness, the definition of “shortness” is subjective and the length of the short texts is dataset dependent. That is, the single-topic assumption may be too strong for some datasets. To address this limitation, we propose to model the topic number as a Poisson distribution, allowing each short text to be associated with a small number of topics (e.g., one to three topics). This model is named PDMM. Second, DMM (and also PDMM) does not have access to background knowledge (e.g., semantic relations between words) when modeling short texts. When a human being interprets a piece of short text, the understanding is not solely based on its content words, but also their semantic relations. Recent advances in word embeddings offer effective learning of word semantic relations from a large corpus. Such auxiliary word embeddings enable us to address the second limitation. To this end, we propose to promote the semantically related words under the same topic during the sampling process, by using the generalized Pólya urn (GPU) model. Through the GPU model, background knowledge about word semantic relations learned from millions of external documents can be easily exploited to improve topic modeling for short texts. By directly extending the PDMM model with the GPU model, we propose two more effective topic models for short texts, named GPU-DMM and GPU-PDMM. Through extensive experiments on two real-world short text collections in two languages, we demonstrate that PDMM achieves better topic representations than state-of-the-art models, measured by topic coherence. The learned topic representation leads to better accuracy in a text classification task, as an indirect evaluation. Both GPU-DMM and GPU-PDMM further improve topic coherence and text classification accuracy. GPU-PDMM outperforms GPU-DMM at the price of higher computational costs.

Over the past few decades, the world has witnessed an unprecedented explosion of information. Of these, a substantial portion consists of unlabeled textual data, such as tweets, news articles, product reviews, and web snippets.
\nAs labeling is extremely expensive, time-consuming, and sometimes biased,
\nhow to effectively analyze these data becomes an imperative.
\nOwing to this, Neural Topic Models (NTMs) have emerged as a promising solution that attracts considerable research attention for their capability and interpretability.
\nThey automatically discover latent topics from unlabeled textual data through neural networks, enabling unsupervised document understanding.
\nThey have derived various downstream applications, such as content recommendation, trend analysis, and text summarization.
\nCompared to conventional topic models like LDA, NTMs offer structural flexibility and also support gradient back-propagation,
\nwhich avoids complicated model-specific derivations and well handle large-scale data.
\n
\nHowever, despite their promise, existing NTMs generally encounter several critical challenges. On the one hand, NTMs often produce low-quality topics, which are even incomparable to conventional models.
\nThese topics tend to be incoherent or repetitive, significantly diminishing their informativeness.
\nOn the other hand, NTMs struggle with low inference ability, leading to less accurate topic distributions for documents.
\nThis limitation greatly hinders document understanding and undermines the subsequent analysis, reasoning, or decision making processes.
\nDue to these challenges, existing NTMs are less useful and applicable for downstream tasks or applications.
\nAs a result, it is necessary to enhance NTMs to deliver more reliable and informative topic modeling.
\n
\nThis thesis aims to advance neural topic modeling by addressing these key challenges.
\nIn particular, we focus on four most popular scenarios: short-texts, cross-lingual, hierarchical, and basic neural topic modeling.
\nFirst, we propose a novel neural topic model tailored for short texts. 
\nThis model leverages a topic-semantic contrastive learning method to captures the similarity relations among short text samples, which works regardless of data augmentation availability.
\nThis refines short text representations, enriches learning signals, thus effectively alleviating the data sparsity issue of short texts and producing informative topics.
\n
\nSecond, we explore dynamic topic modeling.
\nWe present a neural dynamic topic model to track the evolution of dynamic topics by building the contrastive relations among them, rather than relying on the conventional Markov Chains. This model further explicitly excludes unassociated words from dynamic topics to enhance the alignment to their respective time slices.
\nThese improvements enable our model to reliably track topic evolution with high diversity.
\n
\nThird, we focus on the the affinity, rationality, and diversity of hierarchical topic modeling.
\nOur proposed neural hierarchical topic model ensures the sparsity and balance of cross-level topic dependencies using a transport plan dependency method.
\nMoreover, it distributes different semantic granularity to topics at different levels by disentangled decoding.
\nWith these, our model produces affinitive, rational, and diverse topic hierarchies.
\n
\nFourth, we tackle the topic collapsing issue and propose a basic neural topic model.
\nApart from the common reconstruction error, this model introduces a new embedding clustering regularization to force each topic embedding to be the center of a separately aggregated word embedding cluster in the semantic space.
\nThis produces topics with distinct semantics and effectively resolves the topic collapsing issue.
\n
\nFinally, we develop a comprehensive topic modeling toolkit that includes both previous conventional and cutting-edge neural topic models.
\nThis toolkit covers the complete pipelines of topic modeling, such as dataset preprocessing, model training, and evaluation.
\nThese improvements position our toolkit as a valuable resource to accelerate the research and applications of topic models.
\n
\nIn conclusion, this thesis makes significant contributions towards advancing neural topic modeling through multiple models, various scenarios, and a rigorous and comprehensive benchmark toolkit.
\nThese contributions pave the way for the utilization of neural topic modeling in diverse real-world applications.

Emotion detection is crucial in various domains, including psychology, health, social sciences, and marketing. Specifically, in psychology, identifying negative emotions in short Spanish texts, such as tweets, is vital for understanding individuals' emotional states. However, this process is challenging because of factors such as lack of context, cultural nuances, and ambiguous expressions. Although much research on emotion classification in tweets has focused on applications such as crisis analysis, mental health monitoring, and affective computing, most of it has been conducted in English, leaving a significant gap in addressing the emotional needs of Spanish-speaking communities. To address this gap, we used a corpus of 12,000 Spanish tweets tagged with Ekman's negative emotions (sadness, anger, fear, and disgust). Traditional features (n-grams of different types and sizes), syntactic n-grams, and combined features were evaluated. Different deep neural networks, including convolutional neural networks, Bidirectional Encoder Representations of Transformers (BERT), and the robust optimized BERT approach called RoBERTa, were implemented and compared with traditional machine learning methods to identify the most effective method. Extensive testing revealed that BERT achieved the best result, with a macro F1 score of 0.9973. Furthermore, we reported the carbon emissions generated during the training of each implemented method. This study makes a unique contribution by focusing on negative emotions in Spanish, leveraging one of the largest and highest-quality corpora available. It stands out for implementing advanced transformers such as RoBERTa and integrating combined and syntactic n-grams in traditional methods. Furthermore, it highlights how parameters, features, and preprocessing significantly influence performance.

With the rapid development of information science and social networks, the Internet has accumulated various data containing valuable information and topics. The topic model has become one of the primary semantic modeling and classification methods. It has been widely studied in academia and industry. However, most topic models only focus on long texts and often suffer from semantic sparsity problems. The sparse, short text content and irregular data have brought major challenges to the application of topic models in semantic modeling and topic discovery. To overcome these challenges, researchers have explored topic models and achieved excellent results. However, most of the current topic models are applicable to a specific model task. The majority of current reviews ignore the whole-cycle perspective and framework. It brings great challenges for novices to learn topic models. To deal with the above challenges, we investigate more than a hundred papers on topic models and summarize the research progress on the entire topic model process, including theory, method, datasets, and evaluation indicator. In addition, we also analyzed the statistical data results of the topic model through experiments and introduced its applications in different fields. The paper provides a whole-cycle learning path for novices. It encourages researchers to give more attention to the topic model algorithm and the theory itself without paying extra attention to understanding the relevant datasets, evaluation methods and latest progress.

The boost of short texts is increasing demands for short text classification. Feature extraction based on topic models, especially those utilizing label information, such as Partially Labeled Dirichlet Allocation (PLDA), has been widely employed in normal text classification. However, the application of topic models in short text classification is still an open challenge due to the shortness and sparse nature of short texts. In this paper, we propose a topic model called Short Text PLDA (ST-PLDA) based on PLDA for short text classification. In ST-PLDA, each short text is restricted to one label and assumed to be generated from a single topic associated with its label and a background topic shared by all texts. Thus, ST-PLDA is expected to discover more compact and discriminative topic representations. Extensive experiments are conducted on real-world short text collections and the promising results demonstrate the superiority of ST-PLDA compared with other state-of-the-art methods.

In the past few years, we have witnessed a rapid development of online social media, from which we can access various short texts. Understanding the topic patterns of these short text is significant. Traditional topic models, like LDA, are not suitable when applied to short text topic analysis due to data sparsity. A lot of efforts have been made to solve this problem. However, there is still significant space to improve the effectiveness of these short text specific methods. In this paper, we proposed a novel word co-occurrence network based method, referred to as biterm pseudo document topic model (BPDTM), which extended the previous biterm topic model(BTM) for short text. We utilized the word co-occurrence network to construct biterm pseudo documents. The proposed model is promising since it represents words with their semantic adjacent biterms and is able to model the corpus-level semantic relation between two words. Besides, BPDTM naturally lengthens the documents, which alleviate the influence for performance exerted by data sparsity. Experiments demonstrated that our model outperformed two baselines, i.e. LDA and BTM, which proved its effectiveness on short text topic modeling task.

With the rapid development of social media and e-commerce on the Internet, many short texts such as instant messages, tweets and product comments have become an important form of Internet information. Compared with regular texts, short texts suffer severe data sparsity and informal format. Because of that, mining latent semantics such as topics in short texts is a critical challenge. There are many limitations directly applying common topic models like Latent Dirichlet Allocation (LDA) to short texts. In result of that, many short text topic models were put forward. The Word Network Topic Model (WNTM) is the typical one among them, which transfers sparse document to word space into dense word to word space and then learn topics from it. However, WNTM has its own limitation. The word co-occurrence matrix of the original document is used to construct the word network. This method is simple and intuitive, which can't express the deep meaning between words and words. To solve this problem, we proposed W2V-WNTM, which uses Word2Vec model instead of word co–occurrence to construct word network. Experiment shows it can describe relationship between words better.

Sentiment analysis is a crucial part of natural language processing (NLP) that categorizes text into positive, negative, or neutral sentiments. This technology helps businesses automatically interpret customer emotions, enabling informed decisions in marketing, product development, and customer support. Traditional machine learning models like Random Forest, Naive Bayes, and Support Vector Machine (SVM) have been commonly used for sentiment classification, especially effective for short and concise texts such as brief customer comments. SVM, in particular, performs well in classifying sentiments in short texts. However, the emergence of advanced large language models (LLMs) like GPT-4 has revolutionized sentiment analysis by capturing more nuanced emotional expressions. GPT-4 excels in understanding complex context, sarcasm, and subtle emotional tones within longer texts, offering multi-dimensional sentiment insights such as satisfaction, frustration, and uncertainty. This makes it more capable than traditional models when analyzing detailed customer reviews or conversations. Comparative studies reveal that while traditional models handle short texts efficiently, GPT-4 surpasses them in analyzing in-depth content with greater precision, recall, and F1 scores. GPT-4 also uniquely identifies mixed sentiments that simpler models often miss. In conclusion, although traditional machine learning approaches remain useful for straightforward sentiment analysis tasks, GPT-4 provides a more sophisticated and comprehensive understanding of customer emotions in complex texts. Integrating GPT-4 into sentiment analysis workflows can significantly enhance the accuracy and richness of insights derived from customer feedback, supporting better business decisions.

Understanding contents in social networks by inferring high-quality latent topics from short texts is a significant task in social analysis, which is challenging because social network contents are usually extremely short, noisy and full of informal vocabularies. Due to the lack of sufficient word co-occurrence instances, well-known topic modeling methods such as LDA and LSA cannot uncover high-quality topic structures. Existing research works seek to pool short texts from social networks into pseudo documents or utilize the explicit relations among these short texts such as hashtags in tweets to make classic topic modeling methods work. In this paper, we explore this problem by proposing a topic model for noisy short texts with multiple relations called MRTM (Multiple Relational Topic Modeling). MRTM exploits both explicit and implicit relations by introducing a document-attribute distribution and a two-step random sampling strategy. Extensive experiments, compared with the state-of-the-art topic modeling approaches, demonstrate that MRTM can alleviate the word co-occurrence sparsity and uncover high-quality latent topics from noisy short texts.

Most of the applications currently deal with short texts but understanding short texts is a difficult task for machines. Most of the natural language processing tools find it difficult to process short texts. The main reasons for the difficulty in understanding short texts include: i. Short texts usually do not follow the syntax of the written language; ii. Short texts do not contain sufficient statistical signals to support recent developments in text mining such as topic modeling; iii. Short texts are ambiguous and noisy and are generated in larger volumes which further increase the difficulty in handling them. This paper focuses on two efficient semantic knowledge based techniques for understanding short texts which include i. Harvesting and Analyzing Semantic Knowledge (HASK) technique and ii. Semantic Enrichment (SE) technique. These two techniques are evaluated on the basis of three parameters which include accuracy of the system on correctly creating the required predictions, average execution time and space requirement. Experimental results show that the HASK technique outperforms the SE technique.

A short text topic modeling method based on integrating Gaussian and Logistic coding networks with pre-trained word embeddings

Place name disambiguation is the task of correctly identifying a place from a set of places sharing a common name. It contributes to tasks such as knowledge extraction, query answering, geographic information retrieval, and automatic tagging. Disambiguation quality relies on the ability to correctly identify and interpret contextual clues, complicating the task for short texts. Here we propose a novel approach to the disambiguation of place names from short texts that integrates two models: entity co-occurrence and topic modeling. The first model uses Linked Data to identify related entities to improve disambiguation quality. The second model uses topic modeling to differentiate places based on the terms used to describe them. We evaluate our approach using a corpus of short texts, determine the suitable weight between models, and demonstrate that a combined model outperforms benchmark systems such as DBpedia Spotlight and Open Calais in terms of F1-score and Mean Reciprocal Rank.