Multiple Contrast Tests for Count Data: Small Sample Approximations and Their Limitations

In randomized clinical trials with a time-to-event outcome, the intervention effect could be quantified by a difference in restricted mean survival time (ΔRMST) between the intervention and control groups, defined as the expected survival duration gain due to the intervention over a fixed follow-up period. In cluster randomized trials (CRTs), social units are randomized to intervention or control groups; the correlation between survival times of the individuals within the same cluster must be taken into account in the statistical analysis. In a previous work, we proposed the use of pseudo-values regression, based on generalized estimating equations (GEEs), for estimating ΔRMST in CRTs. We showed that this method correctly estimated the ΔRMST and controlled the type I error rate in CRTs with at least 50 clusters. Here, we propose methods for CRTs with a small number of clusters (<50). We evaluated the performance of four bias-corrections of the GEE sandwich variance estimator of the intervention effect. We also considered the use of a Student t distribution as an alternative to the normal distribution of the GEE Wald test statistic for testing the intervention effect and constructing the confidence interval. With a simulation study, assuming proportional or non-proportional hazards, we showed that the Student t distribution outperformed the normal distribution in terms of type I error rate, and the Fay and Graubard bias-corrected variance led to an appropriate type I error rate whatever the number of clusters. Therefore, we recommend the use of the Fay and Graubard variance estimator combined with a Student t distribution for the pseudo-values regression to correctly estimate the variance of the intervention effect. Finally, we provide an illustrative analysis of the DEMETER trial evaluating the use of a specific endotracheal tube for subglottic secretion drainage to prevent ventilator-associated pneumonia, by comparing each of the methods considered.

Cluster-randomized trials allow for the evaluation of a community-level or group-/cluster-level intervention. For studies that require a cluster-randomized trial design to evaluate cluster-level interventions aimed at controlling vector-borne diseases, it may be difficult to assess a large number of clusters while performing the additional work needed to monitor participants, vectors, and environmental factors associated with the disease. One such example of a cluster-randomized trial with few clusters was the "efficacy and risk of harms of repeated ivermectin mass drug administrations for control of malaria" trial. Although previous work has provided recommendations for analyzing trials like repeated ivermectin mass drug administrations for control of malaria, additional evaluation of the multiple approaches for analysis is needed for study designs with count outcomes. Using a simulation study, we applied three analysis frameworks to three cluster-randomized trial designs (single-year, 2-year parallel, and 2-year crossover) in the context of a 2-year parallel follow-up of repeated ivermectin mass drug administrations for control of malaria. Mixed-effects models, generalized estimating equations, and cluster-level analyses were evaluated. Additional 2-year parallel designs with different numbers of clusters and different cluster correlations were also explored. Mixed-effects models with a small sample correction and unweighted cluster-level summaries yielded both high power and control of the Type I error rate. Generalized estimating equation approaches that utilized small sample corrections controlled the Type I error rate but did not confer greater power when compared to a mixed model approach with small sample correction. The crossover design generally yielded higher power relative to the parallel equivalent. Differences in power between analysis methods became less pronounced as the number of clusters increased. The strength of within-cluster correlation impacted the relative differences in power. Regardless of study design, cluster-level analyses as well as individual-level analyses like mixed-effects models or generalized estimating equations with small sample size corrections can both provide reliable results in small cluster settings. For 2-year parallel follow-up of repeated ivermectin mass drug administrations for control of malaria, we recommend a mixed-effects model with a pseudo-likelihood approximation method and Kenward-Roger correction. Similarly designed studies with small sample sizes and count outcomes should consider adjustments for small sample sizes when using a mixed-effects model or generalized estimating equation for analysis. Although the 2-year parallel follow-up of repeated ivermectin mass drug administrations for control of malaria is already underway as a parallel trial, applying the simulation parameters to a crossover design yielded improved power, suggesting that crossover designs may be valuable in settings where the number of available clusters is limited. Finally, the sensitivity of the analysis approach to the strength of within-cluster correlation should be carefully considered when selecting the primary analysis for a cluster-randomized trial.

Recently, Stewart and Ruberg proposed the use of contrast tests for detecting dose-response relationships. They considered in particular bivariate contrasts for healing rates and gave several possibilities of defining adequate sets of coefficients. This paper extends their work in several directions. First, asymptotic power expressions for both single and multiple contrast tests are derived. Secondly, well known trend tests are rewritten as multiple contrast tests, thus alleviating the inherent problem of choosing adequate contrast coefficients. Thirdly, recent results on the efficient calculation of multivariate normal probabilities overcome the traditional simulation-based methods for the numerical computations. Modifications of the power formulae allow the calculation of sample sizes for given type I and II errors, the spontaneous rate, and the dose-response shape. Some numerical results of a power study for small to moderate sample sizes show that the nominal power is a reasonably good approximation to the actual power. An example from a clinical trial illustrates the practical use of the results.

Considerations of sample size in medical research

Count data are found in Ecological studies. The log-transformed strategy is commonly used in the count or rate data. The rate data are defined as the count data divided by a scale variable such as population at risk or an expected count. The log-transformed strategy is used to satisfy the parametric approach and simplify the model estimation. However, this strategy is not correct. The parameter estimation based on the log-transformed strategy could produce a biased estimate with a high standard error estimate. In this study, we are interested in evaluating the bias of parameter estimates based on the log-transformed strategy on the linear regression model. The generalized linear models have better performance in dealing with count data. However, some practitioners who are more familiar with the linear regression model prefer to use a log-transformed strategy and handle the zero cases by adding small values to zero observations. Simulation data from a Poisson distribution were used to compare the Poisson regression model and the linear regression model combined with the log-transformed strategy. The models were evaluated based on the bias and the root-mean-squared error statistics. We found that the linear regression with log-transformation strategy provided a high bias and a small value of root-mean-squared error, especially for small sample size and a small value of the count data. We also use real data set to explore more detail the uses of log-transformed strategy and compare it with Poisson regression.

Seneta &amp; Chen (2005) tightened the familywise error rate control of Holm's procedure by sharpening its critical values using pairwise dependencies of the $p$-values. In this paper we further sharpen these critical values in the case where the distribution functions of the pairwise maxima of null $p$-values are convex, a property shown to hold in some applications of Holm's procedure. The newer critical values are uniformly larger, providing tighter familywise error rate control than the approach of Seneta &amp; Chen (2005), significantly so under high pairwise positive dependencies. The critical values can be further improved under exchangeable null $p$-values.

Examining a huge amount of data is a typical issue in any research process. However, different statistical processes and techniques play essential role to derive a meaningful conclusion from the presented enormous data. Control of type I error is highly essential for a researcher or statistician while dealing with comparisons test with more than two variables. Multiple testing statistical tests provides a structural system and minimizes the error rate by helping to derive meaningful accurate conclusions. Among the different multiple test procedures Tukey's honestly significant difference test (Tukey's HSD) is most common and popular techniques. The main objective of this study was to explore how significantly selection of confidence level or error rate can affect the rate of committing type I error while drawing conclusion. The effect of committing type I error with selection of confidence level or error rate was explored with citing suitable case study in a special education setting. The case study focuses cognitive effect of music on growth and enhancement of the motor behavior (running, jumping and sliding) of children with mild intellectual disability enrolled in the special school setting. ANOVA test was performed and the significance of selection of individual confidence level and simultaneous confidence level) in Tukey’s HSD test was described.

BackgroundIn microarray gene expression profiling experiments, differentially expressed genes (DEGs) are detected from among tens of thousands of genes on an array using statistical tests. It is important to control the number of false positives or errors that are present in the resultant DEG list. To date, more than 20 different multiple test methods have been reported that compute overall Type I error rates in microarray experiments. However, these methods share the following dilemma: they have low power in cases where only a small number of DEGs exist among a large number of total genes on the array.ResultsThis study contrasts parallel multiplicity of objectively related tests against the traditional simultaneousness of subjectively related tests and proposes a new assessment called the Error Discovery Rate (EDR) for evaluating multiple test comparisons in microarray experiments. Parallel multiple tests use only the negative genes that parallel the positive genes to control the error rate; while simultaneous multiple tests use the total unchanged gene number for error estimates. Here, we demonstrate that the EDR method exhibits improved performance over other methods in specificity and sensitivity in testing expression data sets with sequence digital expression confirmation, in examining simulation data, as well as for three experimental data sets that vary in the proportion of DEGs. The EDR method overcomes a common problem of previous multiple test procedures, namely that the Type I error rate detection power is low when the total gene number used is large but the DEG number is small.ConclusionsMicroarrays are extensively used to address many research questions. However, there is potential to improve the sensitivity and specificity of microarray data analysis by developing improved multiple test comparisons. This study proposes a new view of multiplicity in microarray experiments and the EDR provides an alternative multiple test method for Type I error control in microarray experiments.

An evaluation framework for statistical tests on microarray data

Multiple testing issues have long been considered almost exclusively in the context of General Linear Model, in which usually the significance of a quite limited number of contrasts is tested simultaneously. Most of the procedures used in this context have been designed to control the so-called Family-Wise Error Rate (FWER), defined as the probability of more than one erroneous rejection of a null hypothesis. In the last two decades, large-scale significance tests encountered for example in microarray data analysis have renewed the methodology on multiple testing by introducing novel definitions of Type-I error rates, such as the False Discovery Rate (FDR), to define less conservative procedures. High dimension has also highlighted the need for improvements, to guarantee the control of the error rates in various situations of dependent data. The present article gives motivations for a factor analysis modeling of the covariance between test statistics, both in the situation of simultaneous tests of a small set of contrasts in the General Linear Model and also in high-dimensional significance tests. Impact of the dependence on the power of multiple testing is first discussed and a new procedure controlling the FWER and based on factor-adjusted test statistics is presented as a solution to improve the Type-II error rate with respect to existing methods. Finally, the beneficial impact of the new method is shown on simulated datasets.

This paper proposes a general approach for handling multiple contrast tests for normally distributed data in the presence of partial heteroskedasticity. In contrast to the usual case of complete heteroskedasticity, the treatments belong to subgroups according to their variances. Treatments within these subgroups are homoskedastic, whereas treatments of different subgroups are heteroskedastic. New candidate as well as already existing approaches are described and compared by -simulations. Power simulations show that a gain in power is achieved when the partial heteroskedasticity is taken into account compared to procedures which wrongly assume complete heteroskedasticity. The new approaches will be applied to a phytopathologicalexperiment.

Two goals of multiple-dose factorial trials are (i) demonstrating improved effectiveness of a fixed combination over each of its components as well as (ii) identifying a safe and effective dose range. The authors address both goals though with focus on the second by closure procedures that guarantee strong control of the familywise error rate. Two different families of null hypotheses are investigated for bi-factorial dose-response designs that are monotone with respect to the matrix partial order. One is suitable to find the minimum effective dose(s) and the other one is large enough to identify the highest effective dose step(s). Likelihood ratio tests and appropriate multiple contrast tests are applied to an unbalanced clinical trial example taken from Hung (2000, Statistics in Medicine 19, 2079-2087). Full computer code written in the R language is available from the Internet.

A comprehensive review of group level model performance in the presence of heteroscedasticity: Can a single model control Type I errors in the presence of outliers?

In this study, how the application changes with the size of the scale, type I error rate and power of test values were examined in different sample sizes. The material of this study is the random numbers generated according to different sample sizes and different group means of standard deviation out of populations that hold multinomial distribution. In the study, permutation test, F-test and Kruskal-Wallis (KW) test were examined using combinations with different sample sizes (n = 5, 10, 20, 30, 50) and different p values (probability of occurrence of events which are 0.25, 0.50, 0.75) first interms of Type I error rate and then power of the test using different group averages (Δ =1.0, 1.5, 2.0). As a result of the simulations, it is seen that with small sample sizes, Kruskal-Wallis test was unable to maintain Type I error rate at 0.05 level. In the evaluation of such Likert-type data, it can be stated that, permutation test, one of the distribution free tests, is more practical than other tests in terms of maintaining the Type I error rate at 0.05 level and high power of test values. The permutation test is satisfactory in terms of both the type I error rate and the power of the test. Also, permutation test is a distribution-free test. Therefore, can be used without prerequisites. In almost every combination (π, Δ, k, n), permutation test had similar or superior type 1 error and power values than the F and KW tests. It was observed that compared to the 5 visual analogue scale, when the data is measured in 10 and 20 visual analogue scale, the power of test values decreased. In other words, if more than 5 visual analogue scale are made on the visual analogue scale data, the results would negatively be affected.

It is a typical feature of high dimensional data analysis, for example a microarray study, that a researcher allows thousands of statistical tests at a time. All inferences for the tests are determined using the p-values; a smaller p-value than the α-level of the test signifies a statistically significant test. As the number of tests increases, the chance of observing some small p-values is very high even when all null hypotheses are true. Consequently, we make wrong conclusions on the hypotheses. This type of potential problem frequently happens when we test several hypotheses simultaneously, i.e., the multiple testing problem. Adjustment of the p-values can redress the problem that arises in multiple hypothesis testing. P-value adjustment methods control error rates [type I error (i.e. false positive) and type II error (i.e. false negative)] for each hypothesis in order to achieve high statistical power while keeping the overall Family Wise Error Rate (FWER) no larger than α, where α is most often set to 0.05. However, researchers also consider the False Discovery Rate (FDR), or Positive False Discovery Rate (pFDR) instead of the type I error in multiple comparison problems for microarray studies. The methods involved in controlling the FDR always provide higher statistical power than the methods involved in controlling the type I error rate while keeping the type II error rate low. In practice, microarray studies involve dependent test statistics (or p-values) because genes can be fully dependent on each other in a complicated biological structure. However, some of the p-value adjustment methods only deal with independent test statistics. Thus, we carry out a simulation study with several methods involved in multiple hypothesis testing.