Adaptive Spectral Methods for Simulation Output Analysis

Performance measures of phylogenetic estimation methods such as accuracy, consistency, and power are an attempt at summarizing an ensemble of a given estimator's behavior. These summaries characterize an ensemble behavior with a single number, leading to a variety of definitions. In particular, the relationships between different performance measures such as accuracy and consistency or accuracy and error depend on the exact definition of these measures. In addition, it is relatively common to use large-sample behavior to infer similar behavior for small samples. In fact, large-sample results such as the claimed asymptotic efficiency of the maximum-likelihood estimator are often uninformative for small samples. Conversely, small-sample behavior using simulations is sometimes used to imply large-sample behavior such as consistency. However, such extrapolation is often difficult. How the performance of a phylogenetic estimator scales with the addition of taxa must be qualified with respect to whether the whole tree is being estimated or a fixed subset of taxa is being estimated. It must also be qualified with respect to how tree models are sampled. Over the ensemble of all possible trees of a given size, the performance of the estimators for the whole tree estimate suffers when the tree size becomes larger. However, under certain models of cladogenesis, the estimate can improve with the addition of taxa. In fact, at all numbers of taxa there are subsets of tree models that are easier to estimate than others. This suggests that with judicious addition or subtraction of taxa we can move from tree models that are more difficult to estimate at one number of taxa to those that are easier to estimate at another number of taxa.

In linear models with several observations per cell, estimates of all contrasts are given whose small and large sample behaviour is analogous to that of the estimate of a shift parameter proposed in [2]. In particular, the asymptotic efficiency of these estimates relative to the standard least squares estimates, as the number of observations in each cell gets large, is shown to be the same as the Pitman efficiency of the Wilcoxon test relative to the $t$-test.

Skart is an automated sequential batch-means procedure for constructing a skewness- and autoregression-adjusted confidence interval (CI) for the steady-state mean of a simulation output process either in discrete time (i.e., using observation-based statistics), or in continuous time (i.e., using time-persistent statistics). Skart delivers a CI designed to satisfy user-specified requirements concerning both the CI's coverage probability and its absolute or relative precision. Skart exploits separate adjustments to the classical batch-means CI to account for the effects on the distribution of the underlying Student's t-statistic arising from skewness and autocorrelation of the batch means. The skewness adjustment is based on a Cornish–Fisher expansion for the classical batch-means t-statistic, and the autocorrelation adjustment is based on a first-order autoregressive approximation to the batch-means autocorrelation function. Skart also delivers a point estimator for the steady-state mean that is approximately free of initialization bias. The associated warm-up period is based on iteratively applying Von Neumann's randomness test to spaced batch means with increasing sizes for each batch and its preceding spacer. In extensive experimentation, Skart compared favorably with its competitors. [Supplementary material is available for this article. Go to the publisher's online edition of IIE Transactions for additional discussion, detailed proofs, etc.]

Predictive biomarkers are essential for personalized medicine since they select the best treatment for a specific patient. However, of all biomarkers that are evaluated, only few are eventually used in clinical practice. Many promising biomarkers may be erroneously abandoned because they are investigated in small studies using standard statistical techniques which can cause small sample bias or lack of power. The standard technique for failure time endpoints is Cox proportional hazards regression with a multiplicative interaction term between binary variables of biomarker and treatment. Properties of this model in small studies have not been evaluated so far, therefore we performed a simulation study to understand its small sample behavior. As a remedy, we applied a Firth correction to the score function of the Cox model and obtained confidence intervals (CI) using a profile likelihood (PL) approach. These methods are generally recommended for small studies of different design. Our results show that a Cox model estimates the biomarker-treatment interaction term and the treatment effect in one of the biomarker subgroups with bias, and overestimates their standard errors. Bias is however reduced and power is increased with Firth correction and PL CIs. Hence, the modified Cox model and PL CI should be used instead of a standard Cox model with Wald based CI in small studies of predictive biomarkers.

Is IPO Underperformance a Peso Problem?

We summarize an experimental performance evaluation of WASSP and the Heidelberger-Welch (HW) algorithm, two sequential spectral procedures for steady-state simulation analysis. Both procedures approximate the log-smoothed-periodogram of the batch means after suitable data-truncation to eliminate the effects of initialization bias, finally delivering a confidence-interval estimator for the mean response that satisfies user-specified half-length and coverage-probability requirements. HW uses a Cramer-von Mises test for initialization bias based on the method of standardized time series; and then HW fits a quadratic polynomial to the batch-means log-spectrum. In contrast WASSP uses the von Neumann randomness test and the Shapiro-Wilk normality test to obtain an approximately stationary Gaussian batch-means process whose log-spectrum is approximated via wavelets. Moreover, unlike HW, WASSP estimates the final sample size required to satisfy the user's confidence-interval requirements. Regarding closeness of conformance to both confidence-interval requirements, we found that WASSP outperformed HW in the given test problems.

Estimation of Correlation Coefficient to Correct Selection Bias in Small Samples

Analyzing Experimental Data Using Regression: When is Bias a Practical Concern?

In a dynamic panel data model, the number of moment conditions increases rapidly with the time dimension, resulting in a large dimensional covariance matrix of the instruments. As a consequence, the generalized method of moments (GMM) estimator exhibits a large bias in small samples, especially when the autoregressive parameter is close to unity. To address this issue, we propose a regularized version of the one-step GMM estimator using three regularization schemes based on three different ways of inverting the covariance matrix of the instruments. Under double asymptotics, we show that our regularized estimators are consistent and asymptotically normal. These regularization schemes involve a tuning or regularization parameter which needs to be chosen. We derive a data-driven selection of this regularization parameter based on an approximation of the higher-order mean square error and show its optimality. As an empirical application, we estimate a model of income dynamics.

This paper considers estimation of the parameters for the fractionally integrated class of processes known as ARFIMA. We consider the small sample properties of a conditional sum-of-squares estimator that is asymptotically equivalent to MLE. This estimator has the advantage of being relatively simple and can estimate all the parameters, including the mean, simultaneously. The simulation evidence we present indicates that estimation of the mean can make a considerable difference to the small sample bias and MSE of the other parameter estimates.

Adjusted quasi-maximum likelihood estimator for mixed regressive, spatial autoregressive model and its small sample bias

A bivariate extreme value distribution with fixed marginals is generated by a one-dimensional map called a dependence function. This paper proposes a new nonparametric estimator of this function. Its asymptotic properties are examined, and its small-sample behaviour is compared to that of other rank-based and likelihood-based procedures. The new estimator is shown to be uniformly, strongly convergent and asymptotically unbiased. Through simulations, it is also seen to perform reasonably well against the maximum likelihood estimator based on the correct model and to have smaller L1, L2 and L∞ errors than any existing nonparametric alternative. The n½ consistency of the proposed estimator leads to nonparametric estimation of Tawn's (1988) dependence measure that may be used to test independence in small samples.

We demonstrate that all meta-analyses of partial correlations are biased, and yet hundreds of meta-analyses of partial correlation coefficients (PCCs) are conducted each year widely across economics, business, education, psychology, and medical research. To address these biases, we offer a new weighted average, UWLS+3 . UWLS+3 is the unrestricted weighted least squares weighted average that makes an adjustment to the degrees of freedom that are used to calculate partial correlations and, by doing so, renders trivial any remaining meta-analysis bias. Our simulations also reveal that these meta-analysis biases are small-sample biases (n < 200), and a simple correction factor of (n - 2)/(n - 1) greatly reduces these small-sample biases along with Fisher's z. In many applications where primary studies typically have hundreds or more observations, partial correlations can be meta-analyzed in standard ways with only negligible bias. However, in other fields in the social and the medical sciences that are dominated by small samples, these meta-analysis biases are easily avoidable by our proposed methods.

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

ADHD classification using auto-encoding neural network and binary hypothesis testing