Likelihood inference in some finite mixture models

WINNER OF THE 2007 DEGROOT PRIZE! The prominence of finite mixture modelling is greater than ever. Many important statistical topics like clustering data, outlier treatment, or dealing with unobserved heterogeneity involve finite mixture models in some way or other. The area of potential applications goes beyond simple data analysis and extends to regression analysis and to non-linear time series analysis using Markov switching models. For more than the hundred years since Karl Pearson showed in 1894 how to estimate the five parameters of a mixture of two normal distributions using the method of moments, statistical inference for finite mixture models has been a challenge to everybody who deals with them. In the past ten years, very powerful computational tools emerged for dealing with these models which combine a Bayesian approach with recent Monte simulation techniques based on Markov chains. This book reviews these techniques and covers the most recent advances in the field, among them bridge sampling techniques and reversible jump Markov chain Monte Carlo methods. It is the first time that the Bayesian perspective of finite mixture modelling is systematically presented in book form. It is argued that the Bayesian approach provides much insight in this context and is easily implemented in practice. Although the main focus is on Bayesian inference, the author reviews several frequentist techniques, especially selecting the number of components of a finite mixture model, and discusses some of their shortcomings compared to the Bayesian approach. The aim of this book is to impart the finite mixture and Markov switching approach to statistical modelling to a wide-ranging community. This includes not only statisticians, but also biologists, economists, engineers, financial agents, market researcher, medical researchers or any other frequent user of statistical models. This book should help newcomers to the field to understand how finite mixture and Markov switching models are formulated, what structures they imply on the data, what they could be used for, and how they are estimated. Researchers familiar with the subject also will profit from reading this book. The presentation is rather informal without abandoning mathematical correctness. Previous notions of Bayesian inference and Monte Carlo simulation are useful but not needed.

The mixture model is known as model-based clustering that is used to model a mixture of unknown distributions. The clustering of mixture model is based on four important criteria, including the number of components in the mixture model, clustering kernel (such as Gaussian mixture models, Dirichlet, etc.), estimation methods, and dimensionality (Lai et al., 2019). Finite mixture model is a finite dimensional of a hierarchical model. It is useful in modeling the data with outliers, non-normal distributed or heavy tails. Furthermore, finite mixture model is flexible when fitted with the models that have multiple modes or skewed distribution. The flexibility depends on the increasing number of parameters with the existence of a number of components. The finite mixture model is a flexible model family and widely applied for large heterogeneous datasets. In addition, the finite mixture model is a probabilistic model that is used to examine the presence of unobserved situations or groups and to measure the distinct parameters or distribution. The situations, such as trend, seasoning, crisis time, normal situation, etc., might affect the number of components that exist for a probabilistic distribution. Furthermore, the finite mixture model is essential for time series data because these data exhibit nonlinearity properties and may have missing data or a jump-diffusion situation (Gensler, 2017; McLachlan and Lee, 2019). Keywords: Bayesian method; Finite Mixture Model; Maximum Likelihood Estimation; Prior distribution; Likelihood Function.

We show that the identification results of finite mixture and misclassification models are equivalent in a widely used scenario except for an extra ordering assumption. In the misclassification model, an ordering condition is imposed to pin down the precise values of the latent variable, which are also of interest to researchers and need to be identified. In contrast, finite mixture models are usually identified up to permutations of a latent index, which results in local identification. This local identification is satisfactory because the latent index does not convey any economic meaning. However, reaching global identification is important for estimation, especially when researchers use bootstrap to estimate standard errors. This is because standard errors approximated by bootstrap may be incorrect without a global estimator. We demonstrate that games with multiple equilibria fit in our framework and the global estimator with ordering conditions provides more reliable estimates.

Technological advances led to the generation of large scale complex data. Thus, extraction and retrieval of information to automatically discover latent pattern have been largely studied in the various domains of science and technology. Consequently, machine learning experienced tremendous development and various statistical approaches have been suggested. In particular, data clustering has received a lot of attention. Finite mixture models have been revealed to be one of the flexible and popular approaches in data clustering. Considering mixture models, three crucial aspects should be addressed. The first issue is choosing a distribution which is flexible enough to fit the data. In this paper, a model based on multivariate Beta distributions is proposed. The two other challenges in mixture models are estimation of model's parameters and model complexity. To tackle these challenges, variational inference techniques demonstrated considerable robustness. In this paper, two methods are studied, namely, batch and online variational inferences and the models are evaluated on four medical applications including image segmentation of colorectal cancer, multi‐class colon tissue analysis, digital imaging in skin lesion diagnosis and computer aid detection of Malaria.

Attribute-Level Heterogeneity

Pre-analysis of Multi-batch Bioprocesses Data with Finite Mixture Models in the Reduced Feature Subspace

In the statistical modeling framework, the form of the income distribution can be approaching based on certain statistical distributions. The use of the finite mixture model is relatively flexible in the modeling of the income distribution that has a multimodal pattern. The multimodal pattern can be indicated as the existence of different cluster on the data. The different clusters which can reflect the economic homogeneity of income are represented by the mixture components of the finite mixture model. In this paper, the finite mixture model is implemented for modeling the distribution of household income per capita in Indonesia based on The Fifth Wave of the Indonesia Family Life Survey (IFLS5) 2014-2015. The mixture components of the finite mixture model have been build based on the heavy-tailed statistical distributions, i.e., gamma, lognormal, and Weibull distributions. The estimation of the fitting finite mixture model was conducted using the maximum-likelihood estimation method through the expectation-maximization (EM) algorithm. The suitable finite mixture models were verified with the bootstrap likelihood ratio statistics test, Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). Based on the results, the distribution of household income per capita in Indonesia can be modeled by the four components-lognormal mixture model.

This paper studies the applications of extreme value theory on analysis for closing price data of the Dow-Jones industrial index and Danish fire insurance claims data. The generalized extreme value (GEV) distribution is considered in analyzing the real data, and the hypothesis testing problem for the shape parameter of GEV distribution is investigated based on a new test statistic—the $$L_q$$ -likelihood ratio ( $$L_q$$ R) statistic. The $$L_q$$ R statistic can be treated as a generalized form of the classical likelihood ratio (LR) statistic. We show that the asymptotic behavior of proposed statistic is characterized by the degree of distortion $$q$$ . For small and modest sample sizes, the $$L_q$$ R statistic is still available when $$q$$ is properly chosen. By simulation studies, the proposed statistic not only performs the asymptotic properties, but also outperforms the classical LR statistic as the sample sizes are modest or even small. Meanwhile, the test power based on the new statistic is also simulated by Monte Carlo methods. At last, the models are diagnosed by graphical methods.

We present a novel methodology for estimating the parameters of a finite mixture model (FMM) based on partially rank‐ordered set (PROS) sampling and use it in a fishery application. A PROS sampling design first selects a simple random sample of fish and creates partially rank‐ordered judgement subsets by dividing units into subsets of prespecified sizes. The final measurements are then obtained from these partially ordered judgement subsets. The traditional expectation–maximization algorithm is not directly applicable for these observations. We propose a suitable expectation–maximization algorithm to estimate the parameters of the FMMs based on PROS samples. We also study the problem of classification of the PROS sample into the components of the FMM. We show that the maximum likelihood estimators based on PROS samples perform substantially better than their simple random sample counterparts even with small samples. The results are used to classify a fish population using the length‐frequency data.

Most of the current QTL mapping methods are developed for the backcross and F2 populations. The inter-crossed F3 population is also popular and has been well devised in genetic study. As the F3 population can provide more recombinants than the backcross and F2 populations, it can facilitate the detection of closely linked QTL. In this paper, we propose a statistical model to estimate the effects and positions of QTL for the F3 population by considering the differences in the genome structure between the F3 and F2 populations. As the genotypes of QTL are usually unknown, the statistical model is a finite mixture model. By treating the mixture model as an incomplete data problem, the EM algorithm is implemented to obtain the maximum likelihood estimates of the parameters. Simulations were used to illustrate the performance of the proposed method and evaluate the relative efficiency in the F3 population (as comparing to that of the F2 population) in QTL mapping under several factors, such as sample size, genetic distance between QTL (and markers), the proportion of trait variance contributed by QTL (i.e. heritability). It is found that the QTL mapping by using the F3 population is more powerful and precise in separating closely linked QTL and in estimating the parameters as compare to the use of the F2 population, especially for small sample size, close genetic distance and low heritability. The paper can help outline a strategy of detecting tightly linked QTL for the advanced intercross populations to improve the resolution of genetic architecture of quantitative traits.

The important role of finite mixture models in the statistical analysis of data is underscored by the ever-increasing rate at which articles on mixture applications appear in the statistical and general scientific literature. The aim of this article is to provide an up-to-date account of the theory and methodological developments underlying the applications of finite mixture models. Because of their flexibility, mixture models are being increasingly exploited as a convenient, semiparametric way in which to model unknown distributional shapes. This is in addition to their obvious applications where there is group-structure in the data or where the aim is to explore the data for such structure, as in a cluster analysis. It has now been three decades since the publication of the monograph by McLachlan & Basford (1988) with an emphasis on the potential usefulness of mixture models for inference and clustering. Since then, mixture models have attracted the interest of many researchers and have found many new and interesting fields of application. Thus, the literature on mixture models has expanded enormously, and as a consequence, the bibliography here can only provide selected coverage.

The finite mixture model estimates regression coefficients distinct in each of the different groups of the dataset endogenously determined by this estimator. In what follows the analysis is extended beyond the mean, estimating the model in the tails of the conditional distribution of the dependent variable within each group. While the clustering reduces the overall heterogeneity, since the model is estimated for groups of similar observations, the analysis in the tails uncovers within groups heterogeneity and/or skewness. By integrating the endogenously determined clustering with the quantile regression analysis within each group, enhances the finite mixture models and focuses on the tail behavior of the conditional distribution of the dependent variable. A Monte Carlo experiment and two empirical applications conclude the analysis. In the well‐known birthweight dataset, the finite mixture model identifies and computes the regression coefficients of different groups, each one with its own characteristics, both at the mean and in the tails. In the family expenditure data, the analysis of within and between groups heterogeneity provides interesting economic insights on price elasticities. The analysis in classes proves to be more efficient than the model estimated without clustering. By extending the finite mixture approach to the tails provides a more accurate investigation of the data, introducing a robust tool to unveil sources of within groups heterogeneity and asymmetry otherwise left undetected. It improves efficiency and explanatory power with respect to the standard OLS‐based FMM.

The foundations of -analysis date back to the 1740s, when Euler introduced the theory of partitions, also referred to as additive analytic number theory. Over the years, the discovery of -calculus applications in fields such as operator theory, combinatory, probability theory, and many others has sparked tremendous interest in this mathematical framework. Mixture distributions are probabilistic models in which a data set is assumed to originate from multiple underlying distributions, each contributing with a certain probability. These distributions are commonly used to model complex data structures more accurately. This paper introduces -finite mixture models as a novel extension of the classical finite mixture family, motivated by recent progress in -calculus and generalized probability distributions. By incorporating a deformation parameter , the proposed mixture models offer enhanced modeling flexibility for a variety of stochastic phenomena. The fundamental distributional and statistical properties of the suggested -mixture models are systematically are explored.

We propose a new method of classifying documents into categories. We define for each category a finite mixture model based on soft clustering of words. We treat the problem of classifying documents as that of conducting statistical hypothesis testing over finite mixture models, and employ the EM algorithm to efficiently estimate parameters in a finite mixture model. Experimental results indicate that our method outperforms existing methods.

This paper expands the standard analysis of female labor supply to permit preference heterogeneity by using a finite mixture model. Using the extended model, we obtain theory consistent results whereas a traditional model produces a negative substitution effect. We use our model to illustrate the labor supply effects of a tax reform, corresponding to 1983–1992 changes in the Swedish income-tax schedule. The results shows an expected reduction in tax revenues of about 17%. Finally, we use Monte Carlo simulations and show that our proposed mixture model is robust towards different misspecifications.