ON THE COVERAGE PROBABILITY OF CONFIDENCE INTERVALS IN REGRESSION AFTER VARIABLE SELECTION

The impact of a Hausman pretest, applied to panel data, on the coverage probability of confidence intervals

In this article, multiple comparison procedures with the average of scale parameters for normal and exponential populations are proposed. A subset selection procedure and a confidence interval procedure are investigated. These procedures will have broad applicability in identifying groups of treatments with smaller than the average, larger than the average and not different from the average variability in various fields. Bonferroni inequality is used to bound the probability of a correct selection and the coverage probability of confidence intervals. The Pearson’s approximation to the sampling distribution of the log chi-square ratio utilized in both subset selection and confidence intervals is shown to be satisfactory compared with the simulation results. Statistical tables to implement these procedures for the case of equal sample sizes are provided for use in practice.

Skart is an automated sequential batch-means procedure for constructing a skewness- and autoregression-adjusted confidence interval (CI) for the steady-state mean of a simulation output process either in discrete time (i.e., using observation-based statistics), or in continuous time (i.e., using time-persistent statistics). Skart delivers a CI designed to satisfy user-specified requirements concerning both the CI's coverage probability and its absolute or relative precision. Skart exploits separate adjustments to the classical batch-means CI to account for the effects on the distribution of the underlying Student's t-statistic arising from skewness and autocorrelation of the batch means. The skewness adjustment is based on a Cornish–Fisher expansion for the classical batch-means t-statistic, and the autocorrelation adjustment is based on a first-order autoregressive approximation to the batch-means autocorrelation function. Skart also delivers a point estimator for the steady-state mean that is approximately free of initialization bias. The associated warm-up period is based on iteratively applying Von Neumann's randomness test to spaced batch means with increasing sizes for each batch and its preceding spacer. In extensive experimentation, Skart compared favorably with its competitors. [Supplementary material is available for this article. Go to the publisher's online edition of IIE Transactions for additional discussion, detailed proofs, etc.]

The sandwich estimator, also known as robust covariance matrix estimator, heteroscedasticity-consistent covariance matrix estimate, or empirical covariance matrix estimator, has achieved increasing use in the econometric literature as well as with the growing popularity of generalized estimating equations. Its virtue is that it provides consistent estimates of the covariance matrix for parameter estimates even when the fitted parametric model fails to hold or is not even specified. Surprisingly though, there has been little discussion of properties of the sandwich method other than consistency. We investigate the sandwich estimator in quasi-likelihood models asymptotically, and in the linear case analytically. We show that under certain circumstances when the quasi-likelihood model is correct, the sandwich estimate is often far more variable than the usual parametric variance estimate. The increased variance is a fixed feature of the method and the price that one pays to obtain consistency even when the parametric model fails or when there is heteroscedasticity. We show that the additional variability directly affects the coverage probability of confidence intervals constructed from sandwich variance estimates. In fact, the use of sandwich variance estimates combined with t-distribution quantiles gives confidence intervals with coverage probability falling below the nominal value. We propose an adjustment to compensate for this fact.

Sequest is a fully sequential procedure that delivers improved point and confidence-interval (CI) estimators for a designated steady-state quantile by exploiting a combination of ideas from batching and sectioning. Sequest incorporates effective methods to do the following: (a) eliminate bias in the sectioning-based point estimator that is caused by initialization of the simulation or an inadequate simulation run length (sample size); and (b) adjust the CI half-length for the effects of skewness or correlation in the batching-based point estimators of the designated quantile. Sequest delivers a CI designed to satisfy user-specified requirements concerning both the CI's coverage probability and its absolute or relative precision. We found that Sequest exhibited good small- and large-sample properties in a preliminary evaluation of the procedure's performance on a suite of problems that includes some problems designed to stress test the procedure.

SummaryBootstrap smoothed (bagged) parameter estimators have been proposed as an improvement on estimators found after preliminary data‐based model selection. A result of Efron in 2014 is a very convenient and widely applicable formula for a delta method approximation to the standard deviation of the bootstrap smoothed estimator. This approximation provides an easily computed guide to the accuracy of this estimator. In addition, Efron considered a confidence interval centred on the bootstrap smoothed estimator, with width proportional to the estimate of this approximation to the standard deviation. We evaluate this confidence interval in the scenario of two nested linear regression models, the full model and a simpler model, and a preliminary test of the null hypothesis that the simpler model is correct. We derive computationally convenient expressions for the ideal bootstrap smoothed estimator and the coverage probability and expected length of this confidence interval. In terms of coverage probability, this confidence interval outperforms the post‐model‐selection confidence interval with the same nominal coverage and based on the same preliminary test. We also compare the performance of the confidence interval centred on the bootstrap smoothed estimator, in terms of expected length, to the usual confidence interval, with the same minimum coverage probability, based on the full model.

Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown. To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data. The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record). Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability. Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties. Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.

We consider a linear regression model with regression parameters (θ1,...,θp) and error variance parameter σ2. Our aim is to find a confidence interval with minimum coverage probability 1 − α for a parameter of interest θ1 in the presence of nuisance parameters (θ2,...,θp,σ2). We consider two confidence intervals, the first of which is the standard confidence interval for θ1 with coverage probability 1 − α. The second confidence interval for θ1 is obtained after a variable selection procedure has been applied to θp. This interval is chosen to be as short as possible subject to the constraint that it has minimum coverage probability 1 − α. The confidence intervals are compared using a risk function that is defined as a scaled version of the expected length of the confidence interval. We show that, subject to certain conditions including that [(dimension of response vector) − p] is small, the second confidence interval is preferable to the first when we anticipate (without being certain) that |θp|/σ is small. This comparison of confidence intervals is shown to be mathematically equivalent to a corresponding comparison of prediction intervals.

A combination of hypothesis testing and confidence interval construction is often used in social and behavioral science studies. Sometimes confidence intervals are computed or reported only if a null hypothesis is rejected, perhaps to see whether the range of values is of practical importance. Sometimes they are constructed or reported only if a null hypothesis is accepted, in order to assess the range of plausible nonnull values due to inadequate power to detect them. Even if always computed, they are interpreted differently, depending on whether the null value is or is not included. Furthermore, many studies in which the null hypothesis is not rejected are never published (the "file drawer" problem). This article discusses the coverage probability of nominal 1- confidence intervals when examining intervals that do or do not cover some specified null value, usually zero. A briefer treatment considers interval coverage when undesirable results are suppressed. The coverage probability of such conditional confidence intervals may be very far from the nominal value. The magnitude of the effect of selection on interval coverage probability and possible resultant biases in inference are illustrated, and discussed in relation to effect sizes of importance in social and behavioral science research and to estimation of effect sizes.

A combination of hypothesis testing and confidence interval construction is often used in social and behavioral science studies. Sometimes confidence intervals are computed or reported only if a null hypothesis is rejected, perhaps to see whether the range of values is of practical importance. Sometimes they are constructed or reported only if a null hypothesis is accepted, in order to assess the range of plausible nonnull values due to inadequate power to detect them. Even if always computed, they are interpreted differently, depending on whether the null value is or is not included. Furthermore, many studies in which the null hypothesis is not rejected are never published (the “file drawer” problem). This article discusses the coverage probability of nominal 1− α confidence intervals when examining intervals that do or do not cover some specified null value, usually zero. A briefer treatment considers interval coverage when undesirable results are suppressed. The coverage probability of such conditional confidence intervals may be very far from the nominal value. The magnitude of the effect of selection on interval coverage probability and possible resultant biases in inference are illustrated, and discussed in relation to effect sizes of importance in social and behavioral science research and to estimation of effect sizes.

The lognormal and Weibull distributions are the most popular distributions for modeling lifetime data. In practical applications, they usually fit the data at hand well. However, their predictions may lead to large differences. The main purpose of the present article is to investigate the impacts of mis-specification between the lognormal and Weibull distributions on the interval estimation of a pth quantile of the distributions for complete data. The coverage probabilities of the confidence intervals (CIs) with mis-specification are evaluated. The results indicate that for both the lognormal and the Weibull distributions, the coverage probabilities are significantly influenced by mis-specification, especially for a small or a large p on lower or upper tail of the distributions. In addition, based on the coverage probabilities with correct and mis-specification, a maxmin criterion is proposed to make a choice between these two distributions. The numerical results indicate that for p ≤ 0.05 and 0.6 ≤ p ≤ 0.8, Weibull distribution is suggested to evaluate CIs of a pth quantile of the distributions, while, for 0.2 ≤ p ≤ 0.5 and p = 0.99, lognormal distribution is suggested to evaluate CIs of a pth quantile of the distributions. Besides, for p = 0.9 and 0.95, lognormal distribution is suggested if the sample size is large enough, while, for p = 0.1, Weibull distribution is suggested if the sample size is large enough. Finally, a simulation study is conducted to evaluate the efficiency of the proposed method.

BackgroundLongitudinal ordinal data are commonly analyzed using a marginal proportional odds model for relating ordinal outcomes to covariates in the biomedical and health sciences. The generalized estimating equation (GEE) consistently estimates the regression parameters of marginal models even if the working covariance structure is misspecified. For small-sample longitudinal binary data, recent studies have shown that the bias of regression parameters may result from the GEE and have addressed the issue by applying Firth’s adjustment for the likelihood score equation to the GEE as if generalized estimating functions were likelihood score functions. In this manuscript, for the proportional odds model for longitudinal ordinal data, the small-sample properties of the GEE were investigated, and a bias-reduced GEE (BR-GEE) was derived.MethodsBy applying the adjusted function originally derived for the likelihood score function of the proportional odds model to the GEE, we produced the BR-GEE. We investigated the small-sample properties of both GEE and BR-GEE through simulation and applied them to a clinical study dataset.ResultsIn simulation studies, the BR-GEE had a bias closer to zero, smaller root mean square error than the GEE with coverage probability of confidence interval near or above the nominal level. The simulation also showed that BR-GEE maintained a type I error rate near or below the nominal level.ConclusionsFor the analysis of longitudinal ordinal data involving a small number of subjects, the BR-GEE is advantageous for obtaining estimates of the regression parameters of marginal proportional odds models.

With the growth of interest in network data across fields, the Exponential Random Graph Model (ERGM) has emerged as the leading approach to the statistical analysis of network data. ERGM parameter estimation requires the approximation of an intractable normalizing constant. Simulation methods represent the state-of-the-art approach to approximating the normalizing constant, leading to estimation by Monte Carlo maximum likelihood (MCMLE). MCMLE is accurate when a large sample of networks is used to approximate the normalizing constant. However, MCMLE is computationally expensive, and may be prohibitively so if the size of the network is on the order of 1,000 nodes (i.e., one million potential ties) or greater. When the network is large, one option is maximum pseudolikelihood estimation (MPLE). The standard MPLE is simple and fast, but generally underestimates standard errors. We show that a resampling method---the parametric bootstrap---results in accurate coverage probabilities for confidence intervals. We find that bootstrapped MPLE can be run in 1/5th the time of MCMLE. We study the relative performance of MCMLE and MPLE with simulation studies, and illustrate the two different approaches by applying them to a network of bills introduced in the United State Senate.

We develop several methods for estimating the treatment effect difference defined as the overall log-odds ratio of favourable response in a multicentre clinical trial comparing two treatments with binary response. A simulation study compares the bias and mean squared error of the point estimates and the exact coverage probabilities of confidence intervals obtained distributions.

To evaluate the performance of a disproportionality design, commonly used for analysis of spontaneous reports data such as the FDA Adverse Event Reporting System database, as a potential analytical method for an adverse drug reaction risk identification system using healthcare data. We tested the disproportionality design in 5 real observational healthcare databases and 6 simulated datasets, retrospectively studying the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding. We estimate how well the method can be expected to identify true effects and discriminate from false findings and explore the statistical properties of the estimates the design generates. The primary measure was the area under the curve (AUC) of the receiver operating characteristic (ROC) curve. For each combination of 4 outcomes and 5 databases, 48 versions of disproportionality analysis (DPA) were carried out and the AUC computed. The majority of the AUC values were in the range of 0.35 < AUC < 0.6, which is considered to be poor predictive accuracy, since the value AUC = 0.5 would be expected from mere random assignment. Several DPA versions achieved AUC of about 0.7 for the outcome Acute Renal Failure within the GE database. The overall highest DPA version across all 20 outcome-database combinations was the Bayesian Information Component method with no stratification by age and gender, using first occurrence of outcome and with assumed time-at-risk equal to duration of exposure + 30 d, but none were uniformly optimal. The relative risk estimates for the negative control drug-event combinations were very often biased either upward or downward by a factor of 2 or more. Coverage probabilities of confidence intervals from all methods were far below nominal. The disproportionality methods that we evaluated did not discriminate true positives from true negatives using healthcare data as they seem to do using spontaneous report data.