Scaled test statistics and robust standard errors for non‐normal data in covariance structure analysis: A Monte Carlo study

In this paper we present two new families of test statistics for studying the problem of goodness-of-fit of some data to a latent class model for dichotomous questions based on phi-divergence measures. We also treat the problem of selecting the best model out of a sequence of nested latent class models. In both problems, we study the asymptotic distribution of the corresponding test statistics, showing that they share the same behavior as the corresponding maximum likelihood test statistic.

The asymptotically distribution free (ADF) method is often used to estimate parameters or test models without a normal distribution assumption on variables, both in covariance structure analysis and in correlation structure analysis. However, little has been done to study the differences in behaviors of the ADF method in covariance versus correlation structure analysis. The behaviors of 3 test statistics frequently used to evaluate structural equation models with nonnormally distributed variables, χ2 test TAGLS and its small-sample variants TYB and TF(AGLS) were compared. Results showed that the ADF method in correlation structure analysis with test statistic TAGLS performs much better at small sample sizes than the corresponding test for covariance structures. In contrast, test statistics TYB and TF(AGLS) under the same conditions generally perform better with covariance structures than with correlation structures. It is proposed that excessively large and variable condition numbers of weight matrices are a cause of poor behavior of ADF test statistics in small samples, and results showed that these condition numbers are systematically increased with substantial increase in variance as sample size decreases. Implications for research and practice are discussed.

A well-known ad-hoc approach to conducting structural equation modeling with missing data is to obtain a saturated maximum likelihood (ML) estimate of the population covariance matrix and then to use this estimate in the complete data ML fitting function to obtain parameter estimates. This 2-stage (TS) approach is appealing because it minimizes a familiar function while being only marginally less efficient than the full information ML (FIML) approach. Additional advantages of the TS approach include that it allows for easy incorporation of auxiliary variables and that it is more stable in smaller samples. The main disadvantage is that the standard errors and test statistics provided by the complete data routine will not be correct. Empirical approaches to finding the right corrections for the TS approach have failed to provide unequivocal solutions. In this article, correct standard errors and test statistics for the TS approach with missing completely at random and missing at random normally distributed data are developed and studied. The new TS approach performs well in all conditions, is only marginally less efficient than the FIML approach (and is sometimes more efficient), and has good coverage. Additionally, the residual-based TS statistic outperforms the FIML test statistic in smaller samples. The TS method is thus a viable alternative to FIML, especially in small samples, and its further study is encouraged.

The maximum likelihood (ML) and the weighted likelihood (WL) estimators are commonly used to obtain proficiency level estimates with pre-calibrated item parameters. Both estimators have the same asymptotic standard error (ASE) that can be easily derived from the expected information function of the test. However, the accuracy of this asymptotic formula is unclear with short tests when only a few items are administered. The purpose of this paper is to compare the ASE of these estimators with their exact values, evaluated at the proficiency-level estimates. The exact standard error ( SE) is computed by generating the full exact sample distribution of the estimators, so its practical feasibility is limited to small tests (except under the Rasch model). A simulation study was conducted to compare the ASE and the exact SE of the ML and WL estimators, with the “true” SE (i.e., computed as the exact SE with the true proficiency levels). It is concluded that with small tests, the exact SEs are less biased and return smaller root mean square error values than the asymptotic SEs, while as expected, the two estimators return similar results with longer tests.

I analyze properties of misspecified discrete choice models and the efficacy of Huber-White (sometimes called ‘robust’) standard errors. The Huber-White correction provides asymptotically correct standard errors for a consistent estimator from a misspecified model. There is little justification for using Huber-White standard errors in discrete choice models since misspecification usually leads to inconsistent estimators. I derive necessary and sufficient conditions for consistency of the maximum likelihood estimator of any potentially misspecified random utility model (e.g. conditional logit). I also derive (easily satisfied) sufficient conditions for consistent estimation of the sign of the data generating parameter. It follows the researcher can consistently test the sign (or nullity) of the parameter from the data generating process using the (possibly) misspecified conditional logit. I investigate small sample properties of the Huber-White estimator via a simulation study and find the correction provides little to no improvement for inferences.

Hypothesis tests

A large-scale simulation design was used to study the sensitivity of maximum likelihood (ML) factor analysis to violations of measurement scale and distributional assumptions in the input data. Product-moment, polychoric. Spearman's rho, and Kendall's tau- b correlations computed from ordinal data were used to estimate a single-factor model. The resulting ML estimates were compared on the bases of convergence rates and improper solutions, accuracy of the loading estimates, fit statistics, and estimated standard errors. The LISREL maximum likelihood solution algorithm was used to estimate model parameters. The polychoric correlation procedure was found to provide the most accurate estimates of pairwise correlations and factor loadings but performed worst on all goodness-of-fit criteria. LISREL overestimated all standard errors, thus not reflecting the effects of standardization as previously assumed. When the data were categorized, the polychoric correlations led to the best estimates of the standard errors.

Poisson regression is used to model the data with the response variable in the form of count data. This modeling must meet the equidispersion assumption. That is, the average value is the same as the variance. However, this assumption is often violated. Violation of the equidispersion assumption in Poisson regression modeling will result in invalid conclusions. These violations are an overdispersion and an underdispersion of the response variable. Generalized Poisson Regression (GPR) is an alternative if there is a violation of the equidispersion assumption. If there are two correlated response variables, modeling will use the Bivariate Generalized Poisson Regression (BGPR). However, in the panel data with the observation unit in the form of an area, BGPR is not quite right because there is spatial and temporal heterogeneity in the data. Geographically and Temporally Weighted Bivariate Generalized Poisson Regression (GTWBGPR) is a method for modeling spatial and temporal heterogeneity data. GTWBGPR is a development of GWBGPR. In GTWBGPR, besides accommodating spatial effects, it also accommodates temporal effects. This research will discuss the parameter estimation and test statistics for the GTWBGPR model. Parameter estimation uses Maximum Likelihood Estimation (MLE), but the result is not closed-form, so it is solved by numerical iteration. The numerical iteration used is Newton-Raphson. The test statistic for simultaneous testing uses the Maximum Likelihood Ratio Test (MLRT). With large samples, then this test statistic has a chi-square distribution approximation. So the test statistic for the partial test uses the Z test statistic.

When the assumption of multivariate normality is violated or when a discrepancy function other than (normal theory) maximum likelihood is used in structural equation models, the null distribution of the test statistic may not be χ2 distributed. Most existing methods to approximate this distribution only match up to 2 moments. In this article, we propose 2 additional approximation methods: a scaled F distribution that matches 3 moments simultaneously and a direct Monte Carlo–based weighted sum of i.i.d. χ2 variates. We also conduct comprehensive simulation studies to compare the new and existing methods for both maximum likelihood and nonmaximum likelihood discrepancy functions and to separately evaluate the effect of sampling uncertainty in the estimated weights of the weighted sum on the performance of the approximation methods.

Covariance structure analysis is used to evaluate hypothesized influences among unmeasured latent and observed variables. As implemented, it is not robust to outliers and bad data. Several robust methods in model fitting and testing are proposed. These include direct estimation of M-estimators of structured parameters and a two-stage procedure based on robust M- and S-estimators of population covariances. The large sample properties of these estimators are obtained. The equivalence between a direct M-estimator and a two-stage estimator based on an M-estimator of population covariance is established when sampling from an elliptical distribution. Two test statistics are presented in judging the adequacy of a hypothesized model; both are asymptotically distribution free if using distribution free weight matrices. So these test statistics possess both finite sample and large sample robustness. The two-stage procedures can be easily adapted into standard software packages by modifying existing asymptotically distribution free procedures. To demonstrate the two-stage procedure, S-estimator and M-estimators under different weight functions are calculated for some real data sets.

The paper develops a two-stage robust procedure for structural equation modeling (SEM) and an R package rsem to facilitate the use of the procedure by applied researchers. In the first stage, M-estimates of the saturated mean vector and covariance matrix of all variables are obtained. Those corresponding to the substantive variables are then fitted to the structural model in the second stage. A sandwich-type covariance matrix is used to obtain consistent standard errors (SE) of the structural parameter estimates. Rescaled, adjusted as well as corrected and F-statistics are proposed for overall model evaluation. Using R and EQS, the R package rsem combines the two stages and generates all the test statistics and consistent SEs. Following the robust analysis, multiple model fit indices and standardized solutions are provided in the corresponding output of EQS. An example with open/closed book examination data illustrates the proper use of the package. The method is further applied to the analysis of a data set from the National Longitudinal Survey of Youth 1997 cohort, and results show that the developed procedure not only gives a better endorsement of the substantive models but also yields estimates with uniformly smaller standard errors than the normal-distribution-based maximum likelihood.

The asymptotically distribution-free (ADF) test statistic for covariance structure analysis (CSA) has been reported to perform very poorly in simulation studies, i.e. it leads to inaccurate decisions regarding the adequacy of models of psychological processes. It is shown in the present study that the poor performance of the ADF test statistic is due to inadequate estimation of the weight matrix (W = gamma -1), which is a critical quantity in the ADF theory. Bootstrap procedures based on Hall's bias reduction perspective are proposed to correct the ADF test statistic. It is shown that the bootstrap correction of additive bias on the ADF test statistic yields the desired tail behaviour as the sample size reaches 500 for a 15-variable-3-factor confirmatory factor-analytic model, even if the distribution of the observed variables is not multivariate normal and the latent factors are dependent. These results help to revive the ADF theory in CSA.

Asymptotically distribution free (ADF) type test statistics play an important role in covariance structure analysis with nonnormal data. However, empirical studies have indicated that ADF-type test statistics rejected the correct models too frequently at all but very large sample sizes. This phenomenon may similarly occur for tests involving nested models. Several statistics related to ADF-type nested tests are given. The new test statistics have the same asymptotic properties as the original ADF-type test statistics. When used to evaluate model restrictions with small sample sizes, our statistics are more conservative in rejecting correct models. Numerical comparison of critical values are given for selected sample sizes and model parameter differences.

A Monte Carlo study was conducted to evaluate the performance of two-group multivariate tests on means under conditions of homogeneous and heterogeneous correlation structure and multivariate normality and nonnormality. The test statistics under study included those based on the work of Hotelling and alternative procedures proposed by James, Yao, Johansen, Nel and van der Merwe, and Kim. Results indicated that the F statistic based on Hotelling's T 2 was outperformed by the alternative procedures which showed good Type I error control under moderate to large sample sizes. Under small sample sizes, the alternative procedures tended to have unacceptable Type I error control. The F statistic based on Hotelling's T 2 is recommended for use when the sample size in each group is small (e.g., n = p + 1) and correlation heterogeneity is mild (<.3), or when groups are approximately equal in size. Though simulated conditions of multivariate nonnormality had some impact on the Type I error rates on some of the procedures, the impact was in general small and apparent only under the smaller sample size conditions.

Over the years several discrepancy functions have been introduced both in the literature and in the software of Structural Equation Modeling (SEM). The test statistics for the discrepancy functions associated with Maximum Likelihood (ML), Generalized Least Squares (GLS), and Normal Theory Weighted Least Squares (NWLS) are all asymptotically equivalent. These test statistics are all approximately distributed as central chi-square under correct model specification and if the observed variables are multivariate normally distributed. However, it is known that the distribution of these test statistics will not approximate a central Chi-square distribution for models containing specification error, but is more likely to follow a non-central Chi-square distribution (Browne 1984). This study investigates the empirical distributions of the ML and NWLS discrepancy functions. The study includes 13 different factor models with different types and degrees of specification error. It is found, except for small samples, that the empirical distribution of the ML-test statistic outperforms the empirical distribution of the NWLS-test statistic in terms of approximation to the theoretical non-central Chi-square distribution. Furthermore, in some cases, it turned out that the non-central Chi-square approximation was not appropriate even for models that contained minor and moderate degrees of specification error.