Using observation-level random effects to model overdispersion in count data in ecology and evolution

Summary The ability to identify key ecological processes is important when solving applied problems. Increasingly, ecologists are adopting Akaike's information criterion (AIC) as a metric to help them assess and select among multiple process‐based ecological models. Surprisingly, however, it is still unclear how best to incorporate AIC into the selection process in order to address the trade‐off between maximizing the probability of retaining the most parsimonious model while minimizing the number of models retained. Ecological count data are often observed to be overdispersed with respect to best‐fitting models. Overdispersion is problematic when performing an AIC analysis, as it can result in selection of overly complex models which can lead to poor ecological inference. This paper describes and illustrates two approaches that deal effectively with overdispersion. The first approach involves modelling the causes of overdispersion implicitly using compound probability distributions. The second approach ignores the causes of overdispersion and uses quasi‐AIC (QAIC) as a metric for model parsimony. Simulations and a novel method that identifies the most parsimonious model are used to demonstrate the utility of the two overdispersion approaches within the context of two ecological examples. The first example addresses binomial data obtained from a study of fish survival (as related to habitat structure) and the second example addresses Poisson data obtained from a study of flower visitation by nectarivores. Applying either overdispersion approach reduces the chance of selecting overly complex models, and both approaches result in very similar ecological inference. In addition, inference can be made more reliable by incorporating model nesting into the selection process (i.e. identifying which models are special cases of others), as it reduces the number of models selected without significantly reducing the probability of retaining the most parsimonious models. Synthesis and applications. When data are overdispersed, inference can be improved by either modelling the causes of overdispersion or applying QAIC as a metric for model parsimony. Inference can also be improved by adopting a model filtering procedure based on how models are nested. The general simulation approach presented in this paper for identifying the most parsimonious model, as defined by information theory, should help to improve our understanding of the reliability of model selection when using AIC, and help the development of better selection rules.

On the credibility of insurance claim frequency: Generalized count models and parametric estimators

This paper proposes a novel two-parameter contiguous zero-inflated distribution derived from the joint distribution of the three-parameter gamma and Poisson distributions, based on Lakshmi's three-parameter gamma distribution. The distribution’s properties and common descriptive statistical measures were derived. The study explores the probability mass function, coefficient of variation, skewedness, and kurtosis in relation to parameter changes and discusses parameter estimation using maximum likelihood and moment-generating function methods. Simulation studies evaluate the consistency and efficiency of the distribution at varying sample sizes and parameters. The model's application to the number of deaths demonstrates its flexibility in capturing excess zeros and over-dispersion in count data, providing a better alternative to some existing zero-inflated models.

Count data are subject to considerable sources of what is often referred to as non-sampling error. Errors such as misclassification, measurement error and unmeasured confounding can lead to substantially biased estimators. It is strongly recommended that epidemiologists not only acknowledge these sorts of errors in data, but incorporate sensitivity analyses into part of the total data analysis. We extend previous work on Poisson regression models that allow for misclassification by thoroughly discussing the basis for the models and allowing for extra-Poisson variability in the form of random effects. Via simulation we show the improvements in inference that are brought about by accounting for both the misclassification and the overdispersion.

Blockwise empirical likelihood for time series of counts

Statistical Methods for Overdispersed Count Data

The negative binomial regression model is a widely adopted approach when dealing with dependent variables that consist of non-negative integers or counts. This model serves as an alternative regression technique for addressing issues related to overdispersion in count data. Typically, the maximum likelihood estimator is employed to estimate the parameters of the negative binomial regression model. However, the maximum likelihood estimator can be highly sensitive to multicollinearity, leading to unreliable results. To eliminate the adverse effects of multicollinearity in the negative binomial regression model, we propose the use of a jackknife version of the Kibria–Lukman estimator. In this study, we conducted a theoretical comparison between the proposed jackknife Kibria–Lukman negative binomial regression estimator and several existing estimators documented in the literature. To assess the performance of the proposed estimator, we conducted two simulation studies and performed a real data application. The results from both the simulation studies and the real data application consistently demonstrated that the proposed jackknife Kibria–Lukman negative binomial regression estimator outperforms other estimators.

We propose a tree-based semi-varying coefficient model for the Conway–Maxwell–Poisson (CMP or COM-Poisson) distribution which is a two-parameter generalization of the Poisson distribution and is flexible enough to capture both under-dispersion and over-dispersion in count data. The advantage of tree-based methods is their scalability to high-dimensional data. We develop CMPMOB, an estimation procedure for a semi-varying coefficient model, using model-based recursive partitioning (MOB). The proposed framework is broader than the existing MOB framework as it allows node-invariant effects to be included in the model. To simplify the computational burden of the exhaustive search employed in the original MOB algorithm, a new split point estimation procedure is proposed by borrowing tools from change point estimation methodology. The proposed method uses only the estimated score functions without fitting models for each split point and, therefore, is computationally simpler. Since the tree-based methods only provide a piece-wise constant approximation to the underlying smooth function, we further propose the CMPBoost semi-varying coefficient model which uses the gradient boosting procedure for estimation. The usefulness of the proposed methods are illustrated using simulation studies and a real example from a bike sharing system in Washington, DC. Supplementary files for this article are available online.

Modeling over-dispersed crash data with a long tail: Examining the accuracy of the dispersion parameter in Negative Binomial models

The Poisson regression model is widely used for count data. This model assumes equidispersion. In practice, equidispersion is seldom reflected in data. However, in real-life data, the variance usually exceeds the mean. This situation is known as overdispersion. Negative binomial distribution and other Poisson mix models are often used to model overdispersion count data. Another extension of the negative binomial distribution in another model for count data is the univariate generalized Waring. In addition, the model developed by Famoye can be used in the analysis of count data. When the count data contains a large number of zeros, it is necessary to use zero-inflated models. In this study, different generalized regression models are emphasized for the analysis of excessive zeros count data. For this purpose, a real data set was analysed with the generalized Poisson model, generalized negative binomial model, generalized negative binomial Famoye, generalized Waring model, and the foregoing zero-inflated models. Log-likelihood, Akaike information criterion, Bayes information criterion, Vuong statistics were used for model comparisons.

ABSTRACTClustered count data are commonly analysed by the generalized linear mixed model (GLMM). Here, the correlation due to clustering and some overdispersion is captured by the inclusion of cluster-specific normally distributed random effects. Often, the model does not capture the variability completely. Therefore, the GLMM can be extended by including a set of gamma random effects. Routinely, the GLMM is fitted by maximizing the marginal likelihood. However, this process is computationally intensive. Although feasible with medium to large data, it can be too time-consuming or computationally intractable with very large data. Therefore, a fast two-stage estimator for correlated, overdispersed count data is proposed. It is rooted in the split-sample methodology. Based on a simulation study, it shows good statistical properties. Furthermore, it is computationally much faster than the full maximum likelihood estimator. The approach is illustrated using a large dataset belonging to a network of Belgian general practices.

Fitting N-mixture models to count data with unmodeled heterogeneity: Bias, diagnostics, and alternative approaches

Molecular ecology regularly requires the analysis of count data that reflect the relative abundance of features of a composition (e.g., taxa in a community, gene transcripts in a tissue). The sampling process that generates these data can be modelled using the multinomial distribution. Replicate multinomial samples inform the relative abundances of features in an underlying Dirichlet distribution. These distributions together form a hierarchical model for relative abundances among replicates and sampling groups. This type of Dirichlet-multinomial modelling (DMM) has been described previously, but its benefits and limitations are largely untested. With simulated data, we quantified the ability of DMM to detect differences in proportions between treatment and control groups, and compared the efficacy of three computational methods to implement DMM-Hamiltonian Monte Carlo (HMC), variational inference (VI), and Gibbs Markov chain Monte Carlo. We report that DMM was better able to detect shifts in relative abundances than analogous analytical tools, while identifying an acceptably low number of false positives. Among methods for implementing DMM, HMC provided the most accurate estimates of relative abundances, and VI was the most computationally efficient. The sensitivity of DMM was exemplified through analysis of previously published data describing lung microbiomes. We report that DMM identified several potentially pathogenic, bacterial taxa as more abundant in the lungs of children who aspirated foreign material during swallowing; these differences went undetected with different statistical approaches. Our results suggest that DMM has strong potential as a statistical method to guide inference in molecular ecology.

Motivated by a study exploring spatiotemporal trends in emergency department use, we develop a class of two-part hurdle models for the analysis of zero-inflated areal count data. The models consist of two components-one for the probability of any emergency department use and one for the number of emergency department visits given use. Through a hierarchical structure, the models incorporate both patient- and region-level predictors, as well as spatially and temporally correlated random effects for each model component. The random effects are assigned multivariate conditionally autoregressive priors, which induce dependence between the components and provide spatial and temporal smoothing across adjacent spatial units and time periods, resulting in improved inferences. To accommodate potential overdispersion, we consider a range of parametric specifications for the positive counts, including truncated negative binomial and generalized Poisson distributions. We adopt a Bayesian inferential approach, and posterior computation is handled conveniently within standard Bayesian software. Our results indicate that the negative binomial and generalized Poisson hurdle models vastly outperform the Poisson hurdle model, demonstrating that overdispersed hurdle models provide a useful approach to analyzing zero-inflated spatiotemporal data.

The Poisson regression model is often used as a first model for count data with covariates. Since this model is a GLM with canonical link, regression parameters can be easily fitted using standard software. However the model requires equidispersion, which might not be valid for the data set under consideration. There have been many models proposed in the literature to allow for overdispersion. One such model is the negative binomial regression model. In addition, score tests have been commonly used to detect overdispersion in the data. However these tests do not allow to quantify the effects of overdispersion. In this paper we propose easily interpretable discrepancy measures which allow to quantify the overdispersion effects when comparing a negative binomial regression to Poisson regression. We propose asymptotic $\alpha$-level tests for testing the size of overdispersion effects in terms of the developed discrepancy measures. A graphical display of p-values curves can then be used to allow for an exact quantification of the overdispersion effects. This can lead to a validation of the Poisson regression or a discrimination of the Poisson regression with respect to the negative binomial regression. The proposed asymptotic tests are investigated in small samples using simulation and applied to two examples.