Structural and Statistical Analysis of Finite Mixture Models Based on q-Calculus

WINNER OF THE 2007 DEGROOT PRIZE! The prominence of finite mixture modelling is greater than ever. Many important statistical topics like clustering data, outlier treatment, or dealing with unobserved heterogeneity involve finite mixture models in some way or other. The area of potential applications goes beyond simple data analysis and extends to regression analysis and to non-linear time series analysis using Markov switching models. For more than the hundred years since Karl Pearson showed in 1894 how to estimate the five parameters of a mixture of two normal distributions using the method of moments, statistical inference for finite mixture models has been a challenge to everybody who deals with them. In the past ten years, very powerful computational tools emerged for dealing with these models which combine a Bayesian approach with recent Monte simulation techniques based on Markov chains. This book reviews these techniques and covers the most recent advances in the field, among them bridge sampling techniques and reversible jump Markov chain Monte Carlo methods. It is the first time that the Bayesian perspective of finite mixture modelling is systematically presented in book form. It is argued that the Bayesian approach provides much insight in this context and is easily implemented in practice. Although the main focus is on Bayesian inference, the author reviews several frequentist techniques, especially selecting the number of components of a finite mixture model, and discusses some of their shortcomings compared to the Bayesian approach. The aim of this book is to impart the finite mixture and Markov switching approach to statistical modelling to a wide-ranging community. This includes not only statisticians, but also biologists, economists, engineers, financial agents, market researcher, medical researchers or any other frequent user of statistical models. This book should help newcomers to the field to understand how finite mixture and Markov switching models are formulated, what structures they imply on the data, what they could be used for, and how they are estimated. Researchers familiar with the subject also will profit from reading this book. The presentation is rather informal without abandoning mathematical correctness. Previous notions of Bayesian inference and Monte Carlo simulation are useful but not needed.

The mixture model is known as model-based clustering that is used to model a mixture of unknown distributions. The clustering of mixture model is based on four important criteria, including the number of components in the mixture model, clustering kernel (such as Gaussian mixture models, Dirichlet, etc.), estimation methods, and dimensionality (Lai et al., 2019). Finite mixture model is a finite dimensional of a hierarchical model. It is useful in modeling the data with outliers, non-normal distributed or heavy tails. Furthermore, finite mixture model is flexible when fitted with the models that have multiple modes or skewed distribution. The flexibility depends on the increasing number of parameters with the existence of a number of components. The finite mixture model is a flexible model family and widely applied for large heterogeneous datasets. In addition, the finite mixture model is a probabilistic model that is used to examine the presence of unobserved situations or groups and to measure the distinct parameters or distribution. The situations, such as trend, seasoning, crisis time, normal situation, etc., might affect the number of components that exist for a probabilistic distribution. Furthermore, the finite mixture model is essential for time series data because these data exhibit nonlinearity properties and may have missing data or a jump-diffusion situation (Gensler, 2017; McLachlan and Lee, 2019). Keywords: Bayesian method; Finite Mixture Model; Maximum Likelihood Estimation; Prior distribution; Likelihood Function.

We show that the identification results of finite mixture and misclassification models are equivalent in a widely used scenario except for an extra ordering assumption. In the misclassification model, an ordering condition is imposed to pin down the precise values of the latent variable, which are also of interest to researchers and need to be identified. In contrast, finite mixture models are usually identified up to permutations of a latent index, which results in local identification. This local identification is satisfactory because the latent index does not convey any economic meaning. However, reaching global identification is important for estimation, especially when researchers use bootstrap to estimate standard errors. This is because standard errors approximated by bootstrap may be incorrect without a global estimator. We demonstrate that games with multiple equilibria fit in our framework and the global estimator with ordering conditions provides more reliable estimates.

Technological advances led to the generation of large scale complex data. Thus, extraction and retrieval of information to automatically discover latent pattern have been largely studied in the various domains of science and technology. Consequently, machine learning experienced tremendous development and various statistical approaches have been suggested. In particular, data clustering has received a lot of attention. Finite mixture models have been revealed to be one of the flexible and popular approaches in data clustering. Considering mixture models, three crucial aspects should be addressed. The first issue is choosing a distribution which is flexible enough to fit the data. In this paper, a model based on multivariate Beta distributions is proposed. The two other challenges in mixture models are estimation of model's parameters and model complexity. To tackle these challenges, variational inference techniques demonstrated considerable robustness. In this paper, two methods are studied, namely, batch and online variational inferences and the models are evaluated on four medical applications including image segmentation of colorectal cancer, multi‐class colon tissue analysis, digital imaging in skin lesion diagnosis and computer aid detection of Malaria.

Pre-analysis of Multi-batch Bioprocesses Data with Finite Mixture Models in the Reduced Feature Subspace

Attribute-Level Heterogeneity

Finite mixture model is a flexible approach for modeling multimodal data. Multimodality can be present in the data when the data constitute several subpopulations. In this study, several two-component mixture distributions for count data are proposed and described to cater for bimodality issue. The distributions considered in developing mixture distributions are Poisson (P), Poisson Lindley (PL), negative binomial (NB) as well as negative binomial Lindley (NBL), and altogether, a total of ten two-component mixture distributions are obtained. The maximum likelihood estimators for each mixture distribution are obtained by employing the L-BFGS-B method. A comparison study based on graphical approach is conducted to investigate the effect of mixing proportion on the resulting mixture distribution which are found based on different shapes of probability curve and positions of the mode. A simulation study is conducted to investigate the performance of each mixture distribution in fitting data that come from two subpopulations with different mean and dispersion values. Three mixture models which are P-NB, PL-NB and NB-NB, are the most commonly identified as adequate in describing the simulated data with various different types of mixing properties. These three distributions are considered to be the most flexible and thus, suggested for real data applications.

Chapter Preview. Actuaries have been studying loss distributions since the emergence of the profession. Numerous studies have found that the widely used distributions, such as lognormal, Pareto, and gamma, do not fit insurance data well. Mixture distributions have gained popularity in recent years because of their flexibility in representing insurance losses from various sizes of claims, especially on the right tail. To incorporate the mixture distributions into the framework of popular generalized linear models (GLMs), the authors propose to use finite mixture models (FMMs) to analyze insurance loss data. The regression approach enhances the traditional whole-book distribution analysis by capturing the impact of individual explanatory variables. FMM improves the standard GLM by addressing distribution-related problems, such as heteroskedasticity, over- and underdispersion, unobserved heterogeneity, and fat tails. A case study with applications on claims triage and on high-deductible pricing using workers’ compensation data illustrates those benefits. Introduction Conventional Large Loss Distribution Analysis Large loss distributions have been extensively studied because of their importance in actuarial applications such as increased limit factor and excess loss pricing (Miccolis, 1977), reinsurance retention and layer analysis (Clark, 1996), high deductible pricing (Teng, 1994), and enterprise risk management (Wang, 2002). Klugman et al. (1998) discussed the frequency, severity, and aggregate loss distributions in detail in their book, which has been on the Casualty Actuarial Society syllabus of exam Construction and Evaluation of Actuarial Models for many years. Keatinge (1999) demonstrated that popular single distributions, including those in Klugman et al. (1998), are not adequate to represent the insurance loss well and suggested using mixture exponential distributions to improve the goodness of fit. Beirlant et al. (2001) proposed a flexible generalized Burr-gamma distribution to address the heavy tail of loss and validated the effectiveness of this parametric distribution by comparing its implied excess-of-loss reinsurance premium with other nonparametric and semi-parametric distributions. Matthys et al. (2004) presented an extreme quantile estimator to deal with extreme insurance losses. Fleming (2008) showed that the sample average of any small data from a skewed population is most likely below the true mean and warned the danger of insurance pricing decisions without considering extreme events. Henry and Hsieh (2009) stressed the importance of understanding the heavy tail behavior of a loss distribution and developed a tail index estimator assuming that the insurance loss possess Pareto-type tails.

SummarySecondary ion mass spectrometry is a popular physical technique that allows learning elemental and chemical compositions of various substances. In some cases, the mass spectrum of the sample studied is not observable without that of the liquid solution in which the investigated substance must be placed. To separate the informative part of the observed mixed spectrum from the irrelevant liquid solution part, we develop an approach that is based on finite mixtures. Finite mixture modelling is a statistical technique that is capable of accounting for various shapes and patterns in data. We discuss a mixture model constructed with bell-shaped discrete component distributions that can be employed in the mass spectrometry framework for extracting the informative part of the mixed spectrum of organic objects. The results for assessing variability in the mass spectrum extracted are derived. The procedure is thoroughly illustrated on simulated data sets and applied to several real life problems that are concerned with complex organic dyes placed in a glycerol liquid matrix. The results obtained have clear chemical interpretation and suggest that the approach proposed is very promising.

This paper studies convergence behavior of latent mixing measures that arise in finite and infinite mixture models, using transportation distances (i.e., Wasserstein metrics). The relationship between Wasserstein distances on the space of mixing measures and f-divergence functionals such as Hellinger and Kullback-Leibler distances on the space of mixture distributions is investigated in detail using various identifiability conditions. Convergence in Wasserstein metrics for discrete measures implies convergence of individual atoms that provide support for the measures, thereby providing a natural interpretation of convergence of clusters in clustering applications where mixture models are typically employed. Convergence rates of posterior distributions for latent mixing measures are established, for both finite mixtures of multivariate distributions and infinite mixtures based on the Dirichlet process.

Variational learning for finite Beta-Liouville mixture models

Unsupervised outlier detection using random subspace and subsampling ensembles of Dirichlet process mixtures

A common and old problem in statistics is the separation of a heterogeneous population into more homogeneous subpopulations. A wide variety of approaches and techniques for tackling this problem now exist. The finite mixture model is one such technique used to analyse a given data set and create natural groups. The main advantage of mixture models is that they offer a very flexible and relatively easy method of fitting to a data set.The main focus of this thesis will be the clustering of data with the aid of mixed variable mixture models. There has been a lot of research and many papers written with regard to the analysis of continuous variables. This thesis, however, will be concerned with the techniques that will be able to analyse data from continuous to mixed variables, a combination of continuous and discrete variables.The expectation-maximization (EM) algorithm of Dempster et al. (1977) will be used in this thesis for the clustering of mixed variable data. As demonstrated effectively in numerous publications, this algorithm solves estimation of the model parameters iteratively under an advantageous property of monotonic convergence. An extension of the EM algorithm, the ECM algorithm introduced by Meng and Rubin (1993), will also be considered in this thesis for itsrole within more complicated mixed mixture models.On the application of mixed mixture models, this thesis will use a joint distribution specified by the conditional distribution of the continuous variables, given the values of the discrete variables times the marginal distribution of the latter. There are three models - naive, logistic and multinomial - considered for the discrete variables and two models - independent and location - considered for the continuous variables. These models are combined to make a total of sixmodels for the analysis of mixed variable data.The program EMM, its design and development will be discussed in this thesis. Many real mixed data sets and some simulations will also be analysed to investigate how these proposed methods will compare with existing methods. Some of these data sets will be transformed so as to create a suitable input for the program and hence provide a comparable output so that comparisons can be made.The program EMM is written in the FORTRAN computer language. This means that it is well organised with each segment arranged in separate sub-routines. This allows for easy modification to any part of the program or alternatively the addition of a new subroutine so as to increase the capabilities of the model.

Finite mixture models are an important tool in the statistical analysis of data, for example in data clustering. The optimal parameters of a mixture model are usually computed by maximizing the log-likelihood functional via the Expectation-Maximization algorithm. We propose an alternative approach based on the theory of Mean Field Games, a class of differential games with an infinite number of agents. We show that the solution of a finite state space multi-population Mean Field Games system characterizes the critical points of the log-likelihood functional for a Bernoulli mixture. The approach is then generalized to mixture models of categorical distributions. Hence, the Mean Field Games approach provides a method to compute the parameters of the mixture model, and we show its application to some standard examples in cluster analysis.

Parametric mixture models are commonly used in applied work, especially empiri- cal economics, where these models are often employed to learn for example about the proportions of various types in a given population. This paper examines the inference question on the proportions (mixing probability) in a simple mixture model in the pres- ence of nuisance parameters when sample size is large. It is well known that likelihood inference in mixture models is complicated due to 1) lack of point identification, and 2) parameters (for example, mixing probabilities) whose true value may lie on the bound- ary of the parameter space. These issues cause the profiled likelihood ratio (PLR) statistic to admit asymptotic limits that differ discontinuously depending on how the true density of the data approaches the regions of singularities where there is lack of point identification. This lack of uniformity in the asymptotic distribution suggests that confidence intervals based on pointwise asymptotic approximations might lead to faulty inferences. This paper examines this problem in details in a finite mixture model and provides possible fixes based on the parametric bootstrap. We examine the performance of this parametric bootstrap in Monte Carlo experiments and apply it to data from Beauty Contest experiments. We also examine small sample inferences and projection methods.