Multiple testing procedures under positive dependency with block structure

Large-scale multiple testing with correlated tests and auxiliary statistics arises in a wide range of scientific fields. Conventional multiple testing procedures largely ignored auxiliary information, such as sparsity information, and the dependence structure among tests. This may result in loss of testing efficiency. In this paper, we propose a procedure, called multivariate local index of significance (mvLIS) procedure, for large-scale multiple testing. The mvLIS procedure can not only characterize local correlations among tests via a Markov chain but also incorporates auxiliary information via multivariate statistics. We present that the oracle mvLIS procedure is valid, namely, it controls false discovery rate (FDR) at the pre-specified level, and show that it yields the smallest false non-discovery rate (FNR) at the same FDR level. Then a data-driven mvLIS procedure is developed to mimic the oracle procedure. Comprehensive simulation studies and a real data analysis of schizophrenia (SCZ) data are performed to illustrate the superior performance of the mvLIS procedure. Moreover, as a byproduct that is of independent interest, we generalize the single-index modulated (SIM) multiple testing procedure, which embeds prior information via 2-dimensional p-values, to allow for d-dimensional ( d ≥ 3 ) statistics in multiple testing. The detailed extension is deferred to Discussion.

Genetic and Microenvironment Features Do Not Distinguish Follicular Lymphoma Patients Requiring Immediate or Deferred Treatment.

Benjamini and Hochberg suggest that the false discovery rate may\nbe the appropriate error rate to control in many applied multiple testing\nproblems. A simple procedure was given there as an FDR controlling procedure\nfor independent test statistics and was shown to be much more powerful than\ncomparable procedures which control the traditional familywise error rate. We\nprove that this same procedure also controls the false discovery rate when the\ntest statistics have positive regression dependency on each of the test\nstatistics corresponding to the true null hypotheses. This condition for\npositive dependency is general enough to cover many problems of practical\ninterest, including the comparisons of many treatments with a single control,\nmultivariate normal test statistics with positive correlation matrix and\nmultivariate $t$. Furthermore, the test statistics may be discrete, and the\ntested hypotheses composite without posing special difficulties. For all other\nforms of dependency, a simple conservative modification of the procedure\ncontrols the false discovery rate. Thus the range of problems for which a\nprocedure with proven FDR control can be offered is greatly increased.

Much effort has been made to improve the famous step up procedure of Benjamini and Hochberg given by linear critical values $\frac{i\alpha}{n}$. It is pointed out by Gavrilov, Benjamini and Sarkar that step down multiple testing procedures based on the critical values $\beta_{i}=\frac{i\alpha}{n+1-i(1-\alpha)}$ still control the false discovery rate (FDR) at the upper bound $\alpha$ under basic independence assumptions. Since that result is no longer true for step up procedures and for step down procedures, if the p-values are dependent, a big discussion about the corresponding FDR starts in the literature. The present paper establishes finite sample formulas and bounds for the FDR and the expected number of false rejections for multiple testing procedures using critical values $\beta_{i}$ under martingale and reverse martingale dependence models. It is pointed out that martingale methods are natural tools for the treatment of local FDR estimators which are closely connected to the present coefficients $\beta_{i}$. The martingale approach also yields new results and further inside for the special basic independence model.

The burgeoning field of genomics has revived interest in multiple testing procedures by raising new methodological and computational challenges. For example, microarray experiments generate large multiplicity problems in which thousands of hypotheses are tested simultaneously. Westfall and Young (1993) propose resampling-based p-value adjustment procedures which are highly relevant to microarray experiments. This article discusses dierent criteria for error control in resampling-based multiple testing, including (a) the family wise error rate of Westfall and Young (1993) and (b) the false discovery rate developed by Benjamini and Hochberg (1995), both from a frequentist viewpoint; and (c) the positive false discovery rate of Storey (2002a), which has a Bayesian motivation. We also introduce our recently developed fast algorithm for implementing the minP adjustment to control family-wise error rate. Adjusted p-values for dierent approaches are applied to gene expression data from two recently published microarray studies. The properties of these procedures for multiple testing are compared.

This study introduces a novel p-value-based multiple testing approach tailored for generalized linear models. Despite the crucial role of generalized linear models in statistics, existing methodologies face obstacles arising from the heterogeneous variance of response variables and complex dependencies among estimated parameters. Our aim is to address the challenge of controlling the false discovery rate (FDR) amidst arbitrarily dependent test statistics. Through the development of efficient computational algorithms, we present a versatile statistical framework for multiple testing. The proposed framework accommodates a range of tools developed for constructing a new model matrix in regression-type analysis, including random row permutations and Model-X knockoffs. We devise efficient computing techniques to solve the encountered non-trivial quadratic matrix equations, enabling the construction of paired p-values suitable for the two-step multiple testing procedure proposed by Sarkar and Tang (Biometrika 109(4): 1149–1155, 2022). Theoretical analysis affirms the properties of our approach, demonstrating its capability to control the FDR at a given level. Empirical evaluations further substantiate its promising performance across diverse simulation settings.

ABSTRACTMaking accurate inference for gene regulatory networks, including inferring about pathway-by-pathway interactions, is an important and difficult task. Motivated by such genomic applications, we consider multiple testing for conditional dependence between subgroups of variables. Under a Gaussian graphical model framework, the problem is translated into simultaneous testing for a collection of submatrices of a high-dimensional precision matrix with each submatrix summarizing the dependence structure between two subgroups of variables.A novel multiple testing procedure is proposed and both theoretical and numerical properties of the procedure are investigated. Asymptotic null distribution of the test statistic for an individual hypothesis is established and the proposed multiple testing procedure is shown to asymptotically control the false discovery rate (FDR) and false discovery proportion (FDP) at the prespecified level under regularity conditions. Simulations show that the procedure works well in controlling the FDR and has good power in detecting the true interactions. The procedure is applied to a breast cancer gene expression study to identify between pathway interactions. Supplementary materials for this article are available online.

During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “H1,1 or H1,2”). However, in some cases, they do not make this inference. Instead, they make separate inferences about each of the individual hypotheses that comprise the joint hypothesis (e.g., H1,1 and H1,2). For example, a researcher might use a Bonferroni correction to adjust their alpha level from the conventional level of 0.050 to 0.025 when testing H1,1 and H1,2, find a significant result for H1,1 (p < 0.025) and not for H1,2 (p > 0.025), and so claim support for H1,1 and not for H1,2. However, these separate individual inferences do not require an alpha adjustment. Only a statistical inference about the union alternative hypothesis “H1,1 or H1,2” requires an alpha adjustment because it is based on “at least one” significant result among the two tests, and so it refers to the familywise error rate. Hence, an inconsistent correction occurs when a researcher corrects their alpha level during multiple testing but does not make an inference about a union alternative hypothesis. In the present article, I discuss this inconsistent correction problem, including its reduction in statistical power for tests of individual hypotheses and its potential causes vis-à-vis error rate confusions and the alpha adjustment ritual. I also provide three illustrations of inconsistent corrections from recent psychology studies. I conclude that inconsistent corrections represent a symptom of statisticism, and I call for a more nuanced inference-based approach to multiple testing corrections.

Identifying influential variables in high-dimensional matrix-valued data while controlling the false discovery rate (FDR) is a critical challenge in modern data science. We propose a novel, model-free procedure specifically designed for simultaneous row and column selection in matrix predictor regression. Our approach utilizes folding selection subspaces (FSS) to formulate structured hypotheses and employs data splitting to construct mirror statistics from FSS estimators. This design bypasses restrictive model specification and the need for p-value computation. Key theoretical contributions include establishing the asymptotic distribution of FSS estimators and proving the mirror statistic is asymptotically symmetric with respect to zero under the null hypothesis. Using the symmetry, we develop a multiple hypothesis testing procedure with data-driven thresholds that provably controls the FDR for row and column at the desired level asymptotically. The framework is further extended to control element-wise FDR under specific structural assumptions. Extensive simulations and a real data analysis demonstrate the superior performance of the proposed method over existing approaches across various settings.

This paper provides an alternative approach to penalized regression for model selection in the context of high‐dimensional linear regressions where the number of covariates is large, often much larger than the number of available observations. We consider the statistical significance of individual covariates one at a time, while taking full account of the multiple testing nature of the inferential problem involved. We refer to the proposed method as One Covariate at a Time Multiple Testing (OCMT) procedure, and use ideas from the multiple testing literature to control the probability of selecting the approximating model, the false positive rate, and the false discovery rate. OCMT is easy to interpret, relates to classical statistical analysis, is valid under general assumptions, is faster to compute, and performs well in small samples. The usefulness of OCMT is also illustrated by an empirical application to forecasting U.S. output growth and inflation.

A Genome-Wide Methylation Study of Severe Vitamin D Deficiency in African American Adolescents

We investigate several distribution-free dependence detection procedures, all based on a shuffling of the trials, from a statistical point of view. The mathematical justification of such procedures lies in the bootstrap principle and its approximation properties. In particular, we show that such a shuffling has mainly to be done on centered quantities-that is, quantities with zero mean under independence-to construct correct p-values, meaning that the corresponding tests control their false positive (FP) rate. Thanks to this study, we introduce a method, named permutation UE, which consists of a multiple testing procedure based on permutation of experimental trials and delayed coincidence count. Each involved single test of this procedure achieves the prescribed level, so that the corresponding multiple testing procedure controls the false discovery rate (FDR), and this with as few assumptions as possible on the underneath distribution, except independence and identical distribution across trials. The mathematical meaning of this assumption is discussed, and it is in particular argued that it does not mean what is commonly referred in neuroscience to as cross-trials stationarity. Some simulations show, moreover, that permutation UE outperforms the trial-shuffling of Pipa and Grün ( 2003 ) and the MTGAUE method of Tuleau-Malot etal. ( 2014 ) in terms of single levels and FDR, for a comparable amount of false negatives. Application to real data is also provided.

Fisher’s exact test and Pearson’s chi-squared test are frequently used for testing associations of two binary variables in 2-by-2 contingency tables. In the single test setting, many studies have shown that the asymptotic Pearson’s chi-squared test cannot preserve the test size for small samples and Fisher Exact test tends to be overly conservative. Multiple unconditional exact tests were proposed for small samples as they perform better than the commonly used chi-square and Fisher’s exact test. No comparison of these approaches have been done in the multiple testing setting.This study examines the performances of two unconditional tests (Boschloo and Z-pooled test statistics are used) with Fisher’s exact test as well as asymptotic Pearson’s chi-squared test in a small sample multiple testing scenario via a simulation study. When testing simultaneously many null hypotheses, Benjamini-Hochberg (BH) procedure is typically applied to control the false discovery rate (FDR). The results show that in terms of sensitivity rate, the performances of Z-pooled and Boschloo Statistic are close to each other; Asymptotic chi-squared test is slightly better than the unconditional exact tests; Fisher’s exact test is the least powerful in all different settings. Boschloo’s test is more computation intensive. Z-pooled test is preferred if running time is a concern.

In a study involving the statistical modeling of potential head injury trials, we explore approaches to trial design that could enhance their power to detect treatment-related effects on clinical outcome. The study uses a survey organized by the European Brain Injury Consortium of over 1,000 head-injured patients to characterize the population from which trial participants can be selected. A variety of models are postulated for the effects of "neuroprotective" treatment on outcome, and their interaction with a range of strategies for targeting patients for inclusion in the trial is evaluated. A very simple strategy of targeting patients with an intermediate prognosis was found to allow a reduction in sample size by 30%, with no reduction in statistical power. This paper illustrates an important methodology for studying the characteristics of competing trial designs.

Testing independence among a number of (ultra) high-dimensional random samples is a fundamental and challenging problem. By arranging $n$ identically distributed $p$-dimensional random vectors into a $p\\times n$ data matrix, we investigate the problem of testing independence among columns under the matrix-variate normal modeling of data. We propose a computationally simple and tuning-free test statistic, characterize its limiting null distribution, analyze the statistical power and prove its minimax optimality. As an important by-product of the test statistic, a ratio-consistent estimator for the quadratic functional of a covariance matrix from correlated samples is developed. We further study the effect of correlation among samples to an important high-dimensional inference problem—large-scale multiple testing of Pearson’s correlation coefficients. Indeed, blindly using classical inference results based on the assumed independence of samples will lead to many false discoveries, which suggests the need for conducting independence testing before applying existing methods. To address the challenge arising from correlation among samples, we propose a “sandwich estimator” of Pearson’s correlation coefficient by de-correlating the samples. Based on this approach, the resulting multiple testing procedure asymptotically controls the overall false discovery rate at the nominal level while maintaining good statistical power. Both simulated and real data experiments are carried out to demonstrate the advantages of the proposed methods.