Distributed Gibbs Sampling for Latent Variable Models

Latent variable (LV) models are increasingly being used in environmental epidemiology as a way to summarize multiple environmental exposures and thus minimize statistical concerns that arise in multiple regression. LV models may be especially useful when multivariate exposures are collected repeatedly over time. LV models can accommodate a variety of assumptions but, at the same time, present the user with many choices for model specification particularly in the case of exposure data collected repeatedly over time. For instance, the user could assume conditional independence of observed exposure biomarkers given the latent exposure and, in the case of longitudinal latent exposure variables, time invariance of the measurement model. Choosing which assumptions to relax is not always straightforward. We were motivated by a study of prenatal lead exposure and mental development, where assumptions of the measurement model for the time-changing longitudinal exposure have appreciable impact on (maximum-likelihood) inferences about the health effects of lead exposure. Although we were not particularly interested in characterizing the change of the LV itself, imposing a longitudinal LV structure on the repeated multivariate exposure measures could result in high efficiency gains for the exposure-disease association. We examine the biases of maximum likelihood estimators when assumptions about the measurement model for the longitudinal latent exposure variable are violated. We adapt existing instrumental variable estimators to the case of longitudinal exposures and propose them as an alternative to estimate the health effects of a time-changing latent predictor. We show that instrumental variable estimators remain unbiased for a wide range of data generating models and have advantages in terms of mean squared error. Copyright © 2017 John Wiley & Sons, Ltd.

Text is a burgeoning data source for psychological researchers, but little methodological research has focused on adapting popular modeling approaches for text to the context of psychological research. One popular measurement model for text, topic modeling, uses a latent mixture model to represent topics underlying a body of documents. Recently, psychologists have studied relationships between these topics and other psychological measures by using estimates of the topics as regression predictors along with other manifest variables. While similar two-stage approaches involving estimated latent variables are known to yield biased estimates and incorrect standard errors, two-stage topic modeling approaches have received limited statistical study and, as we show, are subject to the same problems. To address these problems, we proposed a novel statistical model-supervised latent Dirichlet allocation with covariates (SLDAX)-that jointly incorporates a latent variable measurement model of text and a structural regression model to allow the latent topics and other manifest variables to serve as predictors of an outcome. Using a simulation study with data characteristics consistent with psychological text data, we found that SLDAX estimates were generally more accurate and more efficient. To illustrate the application of SLDAX and a two-stage approach, we provide an empirical clinical application to compare the application of both the two-stage and SLDAX approaches. Finally, we implemented the SLDAX model in an open-source R package to facilitate its use and further study. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Hierarchical Dirichlet Process (HDP) has attracted much attention in the research community of natural language processing. Given a corpus, HDP is able to determine the number of topics automatically, possessing an important feature dubbed nonparametric that overcomes the challenging issue of manually specifying a suitable topic number in parametric topic models, such as Latent Dirichlet Allocation (LDA). Nevertheless, HDP requires a much higher computational cost than LDA for parameter estimation. By taking the advantage of multi-threading, a parallel Gibbs sampling algorithm is proposed to estimate parameters for HDP based on the equivalence between HDP and Gamma-Gamma Poisson Process (G2PP) in terms of the generative process. Unfortunately, the above parallel Gibbs sampling algorithm requires to apply the finite approximation on the number of topics manually (i.e., predefine the topic number), thus can not retain the nonparametric feature of HDP. Another drawback of the above models is the lack of capturing the semantic dependencies between words, because the topic assignment of words is independent with each other. Although some works have been done in phrase-based topic modelling, these existing methods are still limited by either enforcing the entire phrase to share a common topic or requiring much complex and time-consuming phrase mining methods. In this paper, we aim to develop a copula guided parallel Gibbs sampling algorithm for HDP which can adjust the number of topics dynamically and capture the latent semantic dependencies between words that compose a coherent segment. Extensive experiments on real-world datasets indicate that our method achieves low perplexities and high topic coherence scores with a small time cost. In addition, we validate the effectiveness of our method on the modelling of word semantic dependencies by comparing the extracted topical phrases with those learned by state-of-the-art phrase-based baselines.

The expectation-maximization (EM) algorithm can compute the maximum-likelihood (ML) or maximum a posterior (MAP) point estimate of the mixture models or latent variable models such as latent Dirichlet allocation (LDA), which has been one of the most popular probabilistic topic modeling methods in the past decade. However, batch EM has high time and space complexities to learn big LDA models from big data streams. In this paper, we present a fast online EM (FOEM) algorithm that infers the topic distribution from the previously unseen documents incrementally with constant memory requirements. Within the stochastic approximation framework, we show that FOEM can converge to the local stationary point of the LDA's likelihood function. By dynamic scheduling for the fast speed and parameter streaming for the low memory usage, FOEM is more efficient for some lifelong topic modeling tasks than the state-of-the-art online LDA algorithms to handle both big data and big models (aka, big topic modeling) on just a PC.

Addressing posterior collapse by splitting decoders in variational recurrent autoencoders

A massive of text that is generated every minute is increasing dramatically. Therefore, it is more and more important to find an effective model to automatic classify the amount of text. Topic models is the most powerful techniques in text classification. There are many research results in the field of topic model have been published in scholarly journals. The Latent Dirichlet Allocation (LDA) is one of the most popular topic models in text classification. Researchers have proposed many topic evolution models based on LDA to solve some specific problems in applications of text classification. And some joint models which based on topic models combined other algorithms have been studied to enhance the performance of text classification. In this paper, we investigated three categories topic models for text classification and briefly introduced their advantages and disadvantages in the applications of text mining. Also, we introduce the generated process of documents and illustrate the graphical model for each topic models.

Often quantities of interest in psychology cannot be observed directly. These unobservable quantities, known as latent variables, tend to be complex, often multidimensional, constructs. In many cases these constructs are categorical, such that individuals belong to mutually exclusive and exhaustive unobservable subgroups. Latent class analysis (LCA) is a statistical approach to modeling a discrete latent variable using multiple, discrete observed variables as indicators. Examples of latent class variables that appear in the psychology literature include temperament type, substance use behavior, teaching style, stages of change in the transtheoretical model, and latent classes of risk. The first section of this chapter provides a conceptual introduction to the concept of a latent class followed by a technical introduction to the mathematical model, including multiple-groups LCA and LCA with covariates. This is followed by a discussion of parameter restrictions, model selection, and goodness-of-fit. The second section demonstrates LCA using the empirical example of depression subtypes in adolescence. Five latent classes were identified based on responses to eight questionnaire items assessing depression symptoms: Non-depressed (characterized by a low probability of reporting all eight depression symptoms), sad, disliked, sad + disliked, and depressed (characterized by a high probability of reporting all eight depression symptoms). The third section presents longitudinal extensions of the model, including repeated-measures LCA and latent transition analysis (LTA). The empirical example is extended to examine change in depression subtypes over time. The final sections describe recent extensions to the latent class model and areas that merit additional research in the future. Keywords: latent class analysis; latent transition analysis; latent variable model; categorical data

In this paper we extend the latent Dirichlet allocation (LDA) topic model to model facial expression dynamics. Our topic model integrates the temporal information of image sequences through redefining topic generation probability without involving new latent variables or increasing inference difficulties. A collapsed Gibbs sampler is derived for batch learning with labeled training dataset and an efficient learning method for testing data is also discussed. We describe the resulting temporal latent topic model (TLTM) in detail and show how it can be applied to facial expression recognition. Experiments on CMU expression database illustrate that the proposed TLTM is very efficient in facial expression recognition.

A hierarchical latent topic model based on sparse coding

Study DesignThis study is a scoping review.ObjectiveThere is a broad variability in the definition of degenerative cervical myelopathy (DCM) and no standardized set of diagnostic criteria to date.MethodsWe interrogated the Myelopathy.org database, a hand-indexed database of primary clinical studies conducted exclusively on DCM in humans between 2005-2021. The DCM inclusion criteria used in these studies were inputted into 3 topic modeling algorithms: Hierarchical Dirichlet Process (HDP), Latent Dirichlet Allocation (LDA), and BERtopic. The emerging topics were subjected to manual labeling and interpretation.ResultsOf 1676 reports, 120 papers (7.16%) had well-defined inclusion criteria and were subjected to topic modeling. Four topics emerged from the HDP model: disturbance from extremity weakness and motor signs; fine-motor and sensory disturbance of upper extremity; a combination of imaging and clinical findings is required for the diagnosis; and “reinforcing” (or modifying) factors that can aid in the diagnosis in borderline cases. The LDA model showed the following topics: disturbance to the patient is required for the diagnosis; reinforcing factors can aid in the diagnosis in borderline cases; clinical findings from the extremities; and a combination of imaging and clinical findings is required for the diagnosis. BERTopic identified the following topics: imaging abnormality, typical clinical features, range of objective criteria, and presence of clinical findings.ConclusionsThis review provides quantifiable data that only a minority of past studies in DCM provided meaningful inclusion criteria. The items and patterns found here are very useful for the development of diagnostic criteria for DCM.

Extending the traditional discrete choice model by incorporating latent psychological factors can help to better understand the individual’s decision-making process and therefore to yield more reliable part-worth estimates and market share predictions. Several integrated choice and latent variable (ICLV) models which merge the conditional logit model with a structural equation model exist in the literature. They assume homogeneity in the part-worths and use latent variables to model the heterogeneity among the respondents. This paper starts from the mixed logit model that describes the heterogeneity in the part-worths and uses the latent variables to decrease the unexplained part of the heterogeneity. The empirical study presented here shows these ICLV models perform very well with respect to model fit and prediction.

Locally discriminative topic modeling

Measurement of non-random attrition effects on mobility rates using trip diaries data

The high dimensionality of global gene expression profiles, where number of variables (genes) is very large compared to the number of observations (samples), presents challenges that affect generalizability and applicability of microarray analysis. Latent variable modeling offers a promising approach to deal with high-dimensional microarray data. The latent variable model is based on a few latent variables that capture most of the gene expression information. Here, we describe how to accomplish a reduction in dimension by a latent variable methodology, which can greatly reduce the number of features used to characterize microarray data. We propose a general latent variable framework for prediction of predefined classes of samples using gene expression profiles from microarray experiments. The framework consists of (i) selection of smaller number of genes that are most differentially expressed between samples, (ii) dimension reduction using hierarchical clustering, where each cluster partition is identified as latent variable, (iii) discretization of gene expression matrix, (iv) fitting the Rasch item response model for genes in each cluster partition to estimate the expression of latent variable, and (v) construction of prediction model with latent variables as covariates to study the relationship between latent variables and phenotype. Two different microarray data sets are used to illustrate a general framework of the approach. We show that the predictive performance of our method is comparable to the current best approach based on an all-gene space. The method is general and can be applied to the other high-dimensional data problems.

Integrating Big Data Into Evaluation: R Code for Topic Identification and Modeling