Topic Representation using Semantic-Based Patterns

Recently, some statistical topic modeling approaches based on LDA have been applied in the field of supervised document classification, where the model generation procedure incorporates prior knowledge to improve the classification performance. However, these customizations of topic modeling are limited by the cumbersome derivation of a specific inference algorithm for each modification. In this paper, we propose a new supervised topic modeling approach for document classification problems, Neural Labeled LDA (NL-LDA), which builds on the VAE framework, and designs a special generative network to incorporate prior information. The proposed model can support semi-supervised learning based on the manifold assumption and low-density assumption. Meanwhile, NL-LDA has a consistent and concise inference method while semi-supervised learning and predicting. Quantitative experimental results demonstrate our model has outstanding performance on supervised document classification relative to the compared approaches, including traditional statistical and neural topic models. Specially, the proposed model can support both single-label and multi-label document classification. The proposed NL-LDA performs significantly well on semi-supervised classification, especially under a small amount of labeled data. Further comparisons with related works also indicate our model is competitive with state-of-the-art topic modeling approaches on semi-supervised classification.

Topic modeling extracts useful potential topics that reflect market information from massive financial news and is widely used in data mining and economic research. Traditional topic modeling approaches such as Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) lack semantic information, and short texts have feature sparse problems. We develop a topic clustering model based on BERT-LDA joint embedding that takes both contextual semantics and thematic narrative into account. We cluster document embeddings with the HDBSCAN algorithm and utilize a class-based TF-IDF (c-TF-IDF) method to create topic representations. Empirical results show that the BERT-LDA model is competitive compared with traditional and single topic models. It generates coherent topic words that are dissimilar to each other.

This paper presents an investigation of topic modeling in embedding spaces performances in the context of depression assessment. Using the textual content of social media users from the eRisk 2018 dataset, a classification task is performed employing features generated from the Embedded Topic Model. To set contrast with traditional topic modeling, a full comparison with the Latent Dirichlet Allocation model is accomplished. An extensive range of topics and different preprocessing strategies are studied to demonstrate the efficiency of the models. Our results show a noteworthy improvement in the explored task from the application of the novel topic modeling approach.

Purpose: This study aims to propose an unsupervised topic modeling algorithm utilizing contrastive learning to overcome the limitations of existing methods.Methods: The proposed algorithm incorporates contrastive learning into topic modeling and employs an end-to-end learning process. This approach embeds documents, topics, and words into a unified space, identifying their relationships and providing more intuitive topic modeling results. The algorithm was applied to scientific and technical literature data and its performance was evaluated against existing topic modeling methods.Results: The proposed algorithm demonstrated comparable or superior quantitative metrics in terms of consistency and diversity when compared to existing topic modeling approaches on scientific and technical literature datasets. These results indicate that the embeddings learned through contrastive learning effectively capture the relationships between documents and topics.Conclusion: The unsupervised topic modeling algorithm leveraging contrastive learning provides more intuitive results compared to traditional topic modeling approaches and proves to be a valuable tool for large-scale text data analysis. Future research can explore its application in various domains and evaluate the algorithm using additional quantitative and qualitative metrics to further expand its applicability.

Many applications require semantic understanding of short texts, and inferring discriminative and coherent latent topics is a critical and fundamental task in these applications. Conventional topic models largely rely on word co-occurrences to derive topics from a collection of documents. However, due to the length of each document, short texts are much more sparse in terms of word co-occurrences. Recent studies show that the Dirichlet Multinomial Mixture (DMM) model is effective for topic inference over short texts by assuming that each piece of short text is generated by a single topic. However, DMM has two main limitations. First, even though it seems reasonable to assume that each short text has only one topic because of its shortness, the definition of “shortness” is subjective and the length of the short texts is dataset dependent. That is, the single-topic assumption may be too strong for some datasets. To address this limitation, we propose to model the topic number as a Poisson distribution, allowing each short text to be associated with a small number of topics (e.g., one to three topics). This model is named PDMM. Second, DMM (and also PDMM) does not have access to background knowledge (e.g., semantic relations between words) when modeling short texts. When a human being interprets a piece of short text, the understanding is not solely based on its content words, but also their semantic relations. Recent advances in word embeddings offer effective learning of word semantic relations from a large corpus. Such auxiliary word embeddings enable us to address the second limitation. To this end, we propose to promote the semantically related words under the same topic during the sampling process, by using the generalized Pólya urn (GPU) model. Through the GPU model, background knowledge about word semantic relations learned from millions of external documents can be easily exploited to improve topic modeling for short texts. By directly extending the PDMM model with the GPU model, we propose two more effective topic models for short texts, named GPU-DMM and GPU-PDMM. Through extensive experiments on two real-world short text collections in two languages, we demonstrate that PDMM achieves better topic representations than state-of-the-art models, measured by topic coherence. The learned topic representation leads to better accuracy in a text classification task, as an indirect evaluation. Both GPU-DMM and GPU-PDMM further improve topic coherence and text classification accuracy. GPU-PDMM outperforms GPU-DMM at the price of higher computational costs.

Objectives: As the volume of mindfulness research continues amassing exponentially, there have been attempts to review works in various aspects of mindfulness research systematically. The present study provides a scoping review via a topic modelling approach to supplement the overall research synthesis effort. Specifically, the objective is to scope the mindfulness research by identifying topics relevant to mindfulness research using the probabilistic topic modelling approach. Methods: A literature search based on “mindfulness” returned 5947 bibliographical records from the Web of Science Core Collection platform (for records up to 20 October 2017). The combined field of titles and abstracts was subjected to probabilistic topic modelling based on latent Dirichlet allocation (LDA). Results: The optimal number of topics suggested was 106. Further interpretation by the research team resulted in a total of 231 Suggested Terms. The terms were further categorised into Condition/Issue, Construct/Philosophy, Modality, Population/Setting and Research Methodology. Conclusions: The topic modelling process obtained a panoptic view of mindfulness research, providing mindfulness researchers with some indicators regarding the range of topics researched. The outcome of this topic modelling effort has been made available at https://hdl.handle.net/10497/20862.

As the volume of mindfulness research continues amassing exponentially, there have been attempts to review works in various aspects of mindfulness research systematically. The present study provides a scoping review via a topic modelling approach to supplement the overall research synthesis effort. Specifically, the objective is to scope the mindfulness research by identifying topics relevant to mindfulness research using the probabilistic topic modelling approach. A literature search based on “mindfulness” returned 5947 bibliographical records from the Web of Science Core Collection platform (for records up to 20 October 2017). The combined field of titles and abstracts was subjected to probabilistic topic modelling based on latent Dirichlet allocation (LDA). The optimal number of topics suggested was 106. Further interpretation by the research team resulted in a total of 231 Suggested Terms. The terms were further categorised into Condition/Issue, Construct/Philosophy, Modality, Population/Setting and Research Methodology. The topic modelling process obtained a panoptic view of mindfulness research, providing mindfulness researchers with some indicators regarding the range of topics researched. The outcome of this topic modelling effort has been made available at https://hdl.handle.net/10497/20862 .

Regulatory authorities worldwide are developing cybersecurity standards and data protection regulations related to Internet of Things (IoT) devices to protect user information. As smart IoT technologies evolve, the implementation of such regulations and standards is highly crucial for allowing the secure storage, processing, and transfer of data. Given the ongoing research and development of smart IoT technologies across various domains, we aim to investigate whether critical compliance standards are adequately addressed in existing academic research. Through an in-depth analysis utilizing topic modeling and large language model (LLM) approaches, we identified the top five researched areas and their relationship with relevant regulations. We identified that while substantial progress has been made in addressing the technical challenges associated with IoT privacy, there remains an urgent need for research that bridges these solutions with socio-technical systems and compliance. Our research emphasizes the importance of adopting privacy-by-design at the initial stages of IoT implementation. Additionally, we determined research gaps in existing studies and proposed future research directions emphasizing the development of IoT frameworks that are not only technologically robust but also align with evolving legal compliance standards.

This article explores and critically evaluates the potential contribution to discourse studies of topic modelling, a group of machine learning methods which have been used with the aim of automatically discovering thematic information in large collections of texts. We critically evaluate the utility of the thematic grouping of texts into ‘topics’ emerging from a large collection of online patient comments about the National Health Service (NHS) in England. We take two approaches to this, one inspired by methods adopted in existing topic modelling research and the other using more established methods of discourse analysis. In the study, we compare the insights produced by each approach and consider the extent to which the automatically generated topics might be of use to discourse analysts attempting to organise and study sizeable datasets. We found that the topic modelling approach was able to group texts into ‘topics’ that were truly thematically coherent with a mixed degree of success, while the more traditional approach to discourse analysis consistently provided a more nuanced perspective on the data which was ultimately closer to the ‘reality’ of the texts it contains. This study thus highlights issues concerning the use of topic modelling and offers recommendations and caveats to researchers employing such approaches to studying discourse in the future.

Objective: The objective is to develop an ensemble of machine learning algorithms to identify multilingual, online articles that are relevant to biosurveillance. Language morphology varies widely across languages and must be accounted for when designing algorithms. Here, we compare the performance of a word embedding-based approach and a topic modeling approach with machine learning algorithms to determine the best method for Chinese, Arabic, and French languages.Introduction: Global biosurveillance is an extremely important, yet challenging task. One form of global biosurveillance comes from harvesting open source online data (e.g. news, blogs, reports, RSS feeds). The information derived from this data can be used for timely detection and identification of biological threats all over the world. However, the more inclusive the data harvesting procedure is to ensure that all potentially relevant articles are collected, the more data that is irrelevant also gets harvested. This issue can become even more complex when the online data is in a non-native language. Foreign language articles not only create language-specific issues for Natural Language Processing (NLP), but also add a significant amount of translation costs. Previous work shows success in the use of combinatory monolingual classifiers in specific applications, e.g., legal domain [1]. A critical component for a comprehensive, online harvesting biosurveillance system is the capability to identify relevant foreign language articles from irrelevant ones based on the initial article information collected, without the additional cost of full text retrieval and translation.Methods: The analysis text dataset contains the title and brief description of 3506 online articles in Chinese, Arabic, and French languages from the date range of August, 17, 2016 to July 5, 2017. The NLP article pre-processing steps are language-specific tokenization and stop words removal. We compare two different approaches: word embeddings and topic modeling (Fig. 1). For word embeddings, we first generate word vectors for the data using a pretrained Word2Vec (W2V) model [2]. Subsequently, the word vectors within a document are averaged to produce a single feature vector for the document. Then, we fit a machine learning algorithm (random forest classifier or Support Vector Machine (SVM)) to the training vectors and get predictions for the test documents. For topic modelling, we used a Latent Dirichlet Allocation (LDA) model to generate five topics for all relevant documents [3]. For each new document, the output is the probability measure for the document belonging to these five topics. Here, we classify the new document by comparing the probability measure with a relevancy threshold.Results: The Word2Vec model combined with a random forest classifier outperformed the other approaches across the three languages (Fig. 2); the Chinese model has an 89% F1-score, the Arabic model has 86%, and the French model has 94%. To decrease the chance of calling a potentially relevant article irrelevant, high recall was more important than high precision. In the Chinese model, the Word2Vec with a random forest approach had the highest recall at 98% (Table 1).Conclusions: We present research findings on different approaches of relevance to biosurveillance identification on non-English texts and identify the best performing methods for implementation into a biosurveillance online article harvesting system. Our initial results suggest that the word embeddings model has an advantage over topic modeling, and the random forest classifier outperforms the SVM. Directions for future work will aim to further expand the list of languages and methods to be compared, e.g., n-grams and non-negative matrix factorization. In addition, we will fine-tune the Arabic and French model for better accuracy results.

Recently, some statistical topic modeling approaches have been widely applied in the field of supervised document classification. However, there are few researches on these approaches under label noise, which widely exists in real-world applications. For example, many large-scale datasets are collected from websites or annotated by varying quality human-workers, and then have a few mislabeled items. In this paper, we propose two robust topic models for document classification problems: Smoothed Labeled LDA (SL-LDA) and Adaptive Labeled LDA (AL-LDA). SL-LDA is an extension of Labeled LDA (L-LDA), which is a classical supervised topic model. The proposed model overcomes the shortcoming of L-LDA, i.e., overfitting on noisy labels, through Dirichlet smoothing. AL-LDA is an iterative optimization framework based on SL-LDA. At each iterative procedure, we update the Dirichlet prior, which incorporates the observed labels, by a concise algorithm based on maximizing entropy and minimizing cross-entropy principles. This method avoids identifying the noisy label, which is a common difficulty existing in label noise cleaning algorithms. Quantitative experimental results on noisy completely at random (NCAR) and Multiple Noisy Sources (MNS) settings demonstrate our models have outstanding performance under noisy labels. Specially, the proposed AL-LDA has significant advantages relative to the state-of-the-art topic modeling approaches under massive label noise.

Understanding contents in social networks by inferring high-quality latent topics from short texts is a significant task in social analysis, which is challenging because social network contents are usually extremely short, noisy and full of informal vocabularies. Due to the lack of sufficient word co-occurrence instances, well-known topic modeling methods such as LDA and LSA cannot uncover high-quality topic structures. Existing research works seek to pool short texts from social networks into pseudo documents or utilize the explicit relations among these short texts such as hashtags in tweets to make classic topic modeling methods work. In this paper, we explore this problem by proposing a topic model for noisy short texts with multiple relations called MRTM (Multiple Relational Topic Modeling). MRTM exploits both explicit and implicit relations by introducing a document-attribute distribution and a two-step random sampling strategy. Extensive experiments, compared with the state-of-the-art topic modeling approaches, demonstrate that MRTM can alleviate the word co-occurrence sparsity and uncover high-quality latent topics from noisy short texts.

Locally discriminative topic modeling

With the emerging of massive short texts, e.g., social media posts and question titles from Q&A systems, discovering valuable information from them is increasingly significant for many real-world applications of content analysis. The family of topic modeling can effectively explore the hidden structures of documents through the assumptions of latent topics. However, due to the sparseness of short texts, the existing topic models, e.g., latent Dirichlet allocation, lose effectiveness on them. To this end, an effective solution, namely Dirichlet multinomial mixture (DMM), supposing that each short text is only associated with a single topic, indirectly enriches document-level word co-occurrences. However, DMM is sensitive to noisy words, where it often learns inaccurate topic representations at the document level. To address this problem, we extend DMM to a novel Laplacian Dirichlet Multinomial Mixture (LapDMM) topic model for short texts. The basic idea of LapDMM is to preserve local neighborhood structures of short texts, enabling to spread topical signals among neighboring documents, so as to modify the inaccurate topic representations. This is achieved by incorporating the variational manifold regularization into the variational objective of DMM, constraining the close short texts with similar variational topic representations. To find nearest neighbors of short texts, before model inference, we construct an offline document graph, where the distances of short texts can be computed by the word mover’s distance. We further develop an online version of LapDMM, namely Online LapDMM, to achieve inference speedup on massive short texts. Carrying this implications, we exploit the spirit of stochastic optimization with mini-batches and an up-to-date document graph that can efficiently find approximate nearest neighbors instead. To evaluate our models, we compare against the state-of-the-art short text topic models on several traditional tasks, i.e., topic quality, document clustering and classification. The empirical results demonstrate that our models achieve very significant performance gains over the baseline models.

In the evolving landscape of entrepreneurship, digital technologies have ushered in new possibilities, attracting considerable academic attention. Despite the burgeoning research in Digital Entrepreneurship, the field remains fragmented, warranting a synthesized overview and structured research agenda. Consistently with the above, the paper presents a quantitative mapping of Digital Entrepreneurship through a bibliometric analysis of its publications. The research aims to address the need for a comprehensive, bibliometric overview of the topic, which has been echoed in recently published papers. In order to achieve this goal, we collected data from the Web of Science database, a common and scientifically sound choice in entrepreneurship research. The data were analyzed by applying Latent Dirichlet Allocation and topic modeling, thus providing a unique approach to bibliometric mapping. Topic modeling allows for the processing and analysis of significant amounts of scientific data, thus making it an ideal tool for bibliometric research. We find the field of Digital Entrepreneurship to be rather lively and in rapid development, with several publication outlets, affiliations, and countries contributing to it. We found four main topics to be extracted: the implications of Digital Entrepreneurship for innovation, Digital Entrepreneurship as an enabler for empowerment, the transformation of business models through digitalization, and the surge of digital platforms as entrepreneurial ecosystems. Additionally, we have provided a comprehensive overview of the theoretical lenses used amid the sample and a structured research agenda built upon extant gaps. From a theoretical perspective, the article serves as a starting point for future research on the topic and a comprehensive analysis of its present and past. From a practical perspective, the study is of interest to digital entrepreneurs willing to learn more about the opportunities and challenges provided by the digital landscape.