Variable Selection for Hidden Markov Models with Continuous Variables and Missing Data

The language is a symbolic system unique to human being. The acquisition of language, which has its meanings in the real world, is important for robots to understand the environment and communicate with us in our daily life. This paper proposes a novel approach to establish a fundamental framework for the robots which can understand language through their whole body motions. The proposed framework is composed of three modules: “motion symbol”, “motion language model”, and “natural language model”. In the motion symbol module, motion data are symbolized by Hidden Markov Models (HMMs). Each HMM represents abstract motion patterns. Then the HMMs are defined as motion symbols. The motion language model is stochastically designed for links between motion symbols and words. This model consists of three layers of motion symbols, latent states and words. The connections between the motion symbol and the latent state, and between the latent state and the words are denoted by two kinds of probabilities respectively. One connection is represented by the probability that the motion symbol generates the latent state, and the other connection is represented by the probability that the latent state generates the word. Therefore, the motion language model can connect the motion symbols to the words through the latent state. The natural language model stochastically represents sequences of words. In this paper, a bigram, which is a special case of N-gram model, is adopted as the natural language model. This model has the words as nodes and transitions between two words as edges. Therefore sentence structure is expressed as transitions among words. The integration of the motion language model and natural language model can be implemented by the search computation for sentences corresponding to motions and for motions corresponding to sentences. Especially, the usage of the bigram as the natural language model provides a simple search computation so that appropriate and fast bidirectional computation between the motions and language can be achieved. Our approach makes it possible for humanoid robots not only to interpret motions as sentences but also to generate motions from sentences. The tests by using various motions and words validate our framework for the language acquisition of humanoid robots.

This article discusses the use of association models to detect group differences and the choice of model selection criteria under different conditions. The performances of several commonly used model selection criteria in log-linear modeling are examined using Monte Carlo simulations. The results suggest that no single criterion can play the role of panacea in model selection. L2/ df, normed fit index (NFI) (or 1 - L12/L02), and Akaike's information criterion (AIC) are systematically biased toward models incorporating group differences. The log-likelihood ratio test (LRT), in conjunction with the nested chi-square difference test, is useful for samples with moderate sizes. Bayesian information criterion (BIC) is most reliable among the measures tested. It consistently provides correct information about group differences in association, especially when the sample size is large. Although BIC may give equivocal results under certain conditions, the ambiguities can be mitigated by an inclusion of competing models and a cautious interpretation of any small improvement in BIC.

SummaryThis paper considers a linear regression model with regression parameter vector β. The parameter of interest is θ= aTβ where a is specified. When, as a first step, a data‐based variable selection (e.g. minimum Akaike information criterion) is used to select a model, it is common statistical practice to then carry out inference about θ, using the same data, based on the (false) assumption that the selected model had been provided a priori. The paper considers a confidence interval for θ with nominal coverage 1 ‐ α constructed on this (false) assumption, and calls this the naive 1 ‐ α confidence interval. The minimum coverage probability of this confidence interval can be calculated for simple variable selection procedures involving only a single variable. However, the kinds of variable selection procedures used in practice are typically much more complicated. For the real‐life data presented in this paper, there are 20 variables each of which is to be either included or not, leading to 220 different models. The coverage probability at any given value of the parameters provides an upper bound on the minimum coverage probability of the naive confidence interval. This paper derives a new Monte Carlo simulation estimator of the coverage probability, which uses conditioning for variance reduction. For these real‐life data, the gain in efficiency of this Monte Carlo simulation due to conditioning ranged from 2 to 6. The paper also presents a simple one‐dimensional search strategy for parameter values at which the coverage probability is relatively small. For these real‐life data, this search leads to parameter values for which the coverage probability of the naive 0.95 confidence interval is 0.79 for variable selection using the Akaike information criterion and 0.70 for variable selection using Bayes information criterion, showing that these confidence intervals are completely inadequate.

OF THE THESIS Hidden Markov Models for Heart Rate Variability with Biometric Applications by Michael R. Walker II Master of Science in Electrical Engineering Washington University in St. Louis, May 2011 Research Advisor: Dr. Joseph A. O’Sullivan The utility of hidden Markov models (HMM) for modeling individual heart rate variability (HRV) is presented. Starting with a physiologically based statistical model for HRV from the literature, we justify use of HMMs and present methods for parameterizing the model. The forward-backward algorithm and expectation-maximization algorithm are used to estimate the model and the hidden states for a given observation sequence of inter-beat intervals. Multiple initialization techniques are presented to avoid local maxima. Model order is determined from the data sequence using the Bayesian information criterion. Models are trained on twelve hour recordings. The models are then used to discriminate the identity of an individual using data from a separate set of testing data. For database from 52 individuals, true identity was verified with an equal error rate of roughly 0.36. While initial results do not demonstrate strong performance as a biometric, HMMs are able to capture some individuality in the HRV signal. Consistency in HRV over twelve hour time scales is also demonstrated.

Most model selection procedures use a fixed penalty penalizing an increase in the size of a model. These nonadaptive selection procedures perform well only in one type of situation. For instance, Bayesian information criterion (BIC) with a large penalty performs well for “small” models and poorly for “large” models, and Akaike's information criterion (AIC) does just the opposite. This article proposes an adaptive model selection procedure that uses a data-adaptive complexity penalty based on a concept of generalized degrees of freedom. The proposed procedure, combining the benefit of a class of nonadaptive procedures, approximates the best performance of this class of procedures across a variety of different situations. This class includes many well-known procedures, such as AIC, BIC, Mallows's Cp, and risk inflation criterion (RIC). The proposed procedure is applied to wavelet thresholding in nonparametric regression and variable selection in least squares regression. Simulation results and an asymptotic analysis support the effectiveness of the proposed procedure.

SummaryCase-cohort designs are widely used in large cohort studies to reduce the cost associated with covariate measurement. In many such studies the number of covariates is very large, so an efficient variable selection method is necessary. In this paper, we study the properties of a variable selection procedure using the smoothly clipped absolute deviation penalty in a case-cohort design with a diverging number of parameters. We establish the consistency and asymptotic normality of the maximum penalized pseudo-partial-likelihood estimator, and show that the proposed variable selection method is consistent and has an asymptotic oracle property. Simulation studies compare the finite-sample performance of the procedure with tuning parameter selection methods based on the Akaike information criterion and the Bayesian information criterion. We make recommendations for use of the proposed procedures in case-cohort studies, and apply them to the Busselton Health Study.

Latent profile analysis with nonnormal mixtures: A Monte Carlo examination of model selection using fit indices

Multiple imputation and maximum likelihood estimation (via the expectation‐maximization algorithm) are two well‐known methods readily used for analyzing data with missing values. While these two methods are often considered as being distinct from one another, multiple imputation (when using improper imputation) is actually equivalent to a stochastic expectation‐maximization approximation to the likelihood. In this article, we exploit this key result to show that familiar likelihood‐based approaches to model selection, such as Akaike's information criterion (AIC) and the Bayesian information criterion (BIC), can be used to choose the imputation model that best fits the observed data. Poor choice of imputation model is known to bias inference, and while sensitivity analysis has often been used to explore the implications of different imputation models, we show that the data can be used to choose an appropriate imputation model via conventional model selection tools. We show that BIC can be consistent for selecting the correct imputation model in the presence of missing data. We verify these results empirically through simulation studies, and demonstrate their practicality on two classical missing data examples. An interesting result we saw in simulations was that not only can parameter estimates be biased by misspecifying the imputation model, but also by overfitting the imputation model. This emphasizes the importance of using model selection not just to choose the appropriate type of imputation model, but also to decide on the appropriate level of imputation model complexity.

In quantitative analysis of tissue vascular permeability using Dynamic-Contrast-Enhanced (DCE) MR data analysis, one of the main challenges is to choose the best Pharmacokinetic (PK) model among competing models to describe the behavior of the time trace of contrast agent (CA) concentration. The F-Statistic (F-test), Akaike-Information-Criterion (AIC), Bayesian-Information-Criterion (BIC), and Log-Likelihood-Ratio (LLR) tests are the most widely used classical Model-Selection (MS) techniques. Our group has recently introduced an Adaptive-MS (AMS) method to perform PK-MS in DCE-MRI data. This study investigates the performance of different MS techniques (Classical and Adaptive) for selection of the PK model as well as the impact of the recruited technique on the estimation of PK parameters. In this study, three physiologically Nested Models were used to describe possible physiological conditions of underlying tissue pathology as Model 1, 2, and 3 respectively: The tissue-vascular-compartment is filled with CA with no outward-leakage, with outward-leakage but no evidence of back-flux, and with both outward-and backward-flux. Plasma volume vp, vp, and ktrans, vp, ktrans and kep, are the PK parameters for Models 1, 2 and 3 respectively. 21 Arterial Input Function curves were used to simulate the CA profile. By varying PK parameters (time-resolution = 5.03 sec), 9072 time traces for 12 levels of SNRs were generated. Then confusion matrices were computed for all the MS techniques over 9072 tested hypotheses. The performance of the AMS is superior to the other classical techniques. Other techniques show performances in the following order: LLR, BIC, AICc, AIC and F-Test. All techniques never miss any leaky tissue. Also the AMS generates significantly less-biased estimates of PK parameters compared to the classical techniques while the LLR and BIC methods outperform the other classical techniques. This study suggests that AMS, LLR, and BIC are the best candidates among the MS techniques for PK analysis of DCE-MRI data.

This paper proposed Weighted Schwarz Bayesian Information criterion for the purpose of selecting a best model from various competing models, when heteroscedasticity is present in the survey data. The authors found that the information loss between the true model and fitted models are equally weighted, instead of giving unequal weights. The computation of weights purely depends on the differential entropy of each sample observation and traditional Schwarz Bayesian information criterion was penalized by the weight function which comprised of the Inverse variance to mean ratio (VMR) of the fitted log-quantiles. The weighted Schwarz Bayesian information criterion was proposed in two versions based on the nature of the estimated error variances of the model namely Homogeneous and Heterogeneous WSBIC, respectively. The proposed WSBIC is different from the traditional information criterion of model selection and it leads to conduct a logical statistical treatment for selecting a best model. Finally this procedure was numerically illustrated by fitting 12 different types of stepwise regression models based on 44 independent variables in a BSQ (Bank service Quality) study. Keywords: Schwarz Bayesian information criterion, Weighted Schwarz Bayesian information criterion, differential entropy, log-quantiles, variance to mean ratio

The analysis of the leukemia data from Whitehead/MIT group is a discriminant analysis (also called a supervised learning). Among thousands of genes whose expression levels are measured, not all are needed for discriminant analysis: a gene may either not contribute to the separation of two types of tissues/cancers, or it may be redundant because it is highly correlated with other genes. There are two theoretical frameworks in which variable selection (or gene selection in our case) can be addressed. The first is model selection, and the second is model averaging. We have carried out model selection using Akaike information criterion and Bayesian information criterion with logistic regression (discrimination, prediction, or classification) to determine the number of genes that provide the best model. These model selection criteria set upper limits of 22∼25 and 12∼13 genes for this data set with 38 samples, and the best model consists of only one (no.4847, zyxin) or two genes. We have also carried out model averaging over the best single-gene logistic predictors using three different weights: maximized likelihood, prediction rate on training set, and equal weight. We have observed that the performance of most of these weighted predictors on the testing set is gradually reduced as more genes are included, but a clear cutoff that separates good and bad prediction performance is not found.Key wordsmodel/variable selectionAkaike/Bayesian information criterionmaximum likelihoodlogistic regressiondiscriminant analysis

Bayesian information criterion (BIC) is known to identify the true model consistently as long as the predictor dimension is finite. Recently, its moderate modifications have been shown to be consistent in model selection even when the number of variables diverges. Those works have been done mostly in mean regression, but rarely in quantile regression. The best-known results about BIC for quantile regression are for linear models with a fixed number of variables. In this article, we investigate how BIC can be adapted to high-dimensional linear quantile regression and show that a modified BIC is consistent in model selection when the number of variables diverges as the sample size increases. We also discuss how it can be used for choosing the regularization parameters of penalized approaches that are designed to conduct variable selection and shrinkage estimation simultaneously. Moreover, we extend the results to structured nonparametric quantile models with a diverging number of covariates. We illustrate our theoretical results via some simulated examples and a real data analysis on human eye disease. Supplementary materials for this article are available online.

Unsupervised machine learning via Hidden Markov Models for accurate clustering of plant stress levels based on imaged chlorophyll fluorescence profiles & their rate of change in time

Mixture modeling is a major paradigm for clustering in statistics. In this article, we develop a new block-wise variable selection method for clustering by exploiting the latent states of the hidden Markov model on variable blocks or the Gaussian mixture model. The variable blocks are formed by depth-first-search on a dendrogram created based on the mutual information between any pair of variables. It is demonstrated that the latent states of the variable blocks together with the mixture model parameters can represent the original data effectively and much more compactly. We thus cluster the data using the latent states and select variables according to the relationship between the states and the clusters. As true class labels are unknown in the unsupervised setting, we first generate more refined clusters, namely, semi-clusters, for variable selection and then determine the final clusters based on the dimension reduced data. Experiments on simulated and real data show that the new method is highly competitive in terms of clustering accuracy compared with several widely used methods. Supplementary materials for this article are available online.

We illustrate a recently proposed two-step bootstrap model averaging (bootstrap MA) approach to cope with model selection uncertainty. The predictive performance is investigated in an example and in a simulation study. Results are compared to those derived from other model selection methods. In the framework of the linear regression model we use the two-step bootstrap MA, which consists of a screening step to eliminate covariates thought to have no influence on the response, and a model-averaging step. We also apply the full model, variable selection using backward elimination based on Akaike's Information Criterion (AIC), the Bayes Information Criterion (BIC) and the bagging approach. The predictive performance is measured by the mean squared error (MSE) and the coverage of confidence intervals for the true response. We obtained similar results for all approaches in the example. In the simulation the MSE was reduced by all approaches in comparison to the full model. The smallest values are obtained for bootstrap MA. Only the bootstrap MA and the full model correctly estimated the nominal coverage. The backward elimination procedures led to substantial underestimation and bagging to an overestimation of the true coverage. The screening step of bootstrap MA eliminates most of the unimportant factors. The new bootstrap MA approach shows promising results for predictive performance. It increases practical usefulness by eliminating unimportant factors in the screening step.