Surrogate Data Methods Based on a Shuffling of the Trials for Synchrony Detection: The Centering Issue.

Much effort has been made to improve the famous step up procedure of Benjamini and Hochberg given by linear critical values $\frac{i\alpha}{n}$. It is pointed out by Gavrilov, Benjamini and Sarkar that step down multiple testing procedures based on the critical values $\beta_{i}=\frac{i\alpha}{n+1-i(1-\alpha)}$ still control the false discovery rate (FDR) at the upper bound $\alpha$ under basic independence assumptions. Since that result is no longer true for step up procedures and for step down procedures, if the p-values are dependent, a big discussion about the corresponding FDR starts in the literature. The present paper establishes finite sample formulas and bounds for the FDR and the expected number of false rejections for multiple testing procedures using critical values $\beta_{i}$ under martingale and reverse martingale dependence models. It is pointed out that martingale methods are natural tools for the treatment of local FDR estimators which are closely connected to the present coefficients $\beta_{i}$. The martingale approach also yields new results and further inside for the special basic independence model.

The burgeoning field of genomics has revived interest in multiple testing procedures by raising new methodological and computational challenges. For example, microarray experiments generate large multiplicity problems in which thousands of hypotheses are tested simultaneously. Westfall and Young (1993) propose resampling-based p-value adjustment procedures which are highly relevant to microarray experiments. This article discusses dierent criteria for error control in resampling-based multiple testing, including (a) the family wise error rate of Westfall and Young (1993) and (b) the false discovery rate developed by Benjamini and Hochberg (1995), both from a frequentist viewpoint; and (c) the positive false discovery rate of Storey (2002a), which has a Bayesian motivation. We also introduce our recently developed fast algorithm for implementing the minP adjustment to control family-wise error rate. Adjusted p-values for dierent approaches are applied to gene expression data from two recently published microarray studies. The properties of these procedures for multiple testing are compared.

The classical Benjamini–Hochberg (B-H) method, widely used across various disciplines such as genetics, epidemiology, and social sciences, serves as an established procedure for controlling the false discovery rate (FDR) in multiple comparison scenarios. The B-H method assumes independence among tests, which often does not hold in large-scale dependent datasets. The Benjamini–Yekutieli (B-Y) adjustment controls the FDR under arbitrary dependence but is often very conservative and can lead to a reduction in statistical power. This paper investigates the performance of the B-H and B-Y procedures under specific positive block dependence structures. Two parametric forms of block dependence are considered to model the correlation among paired t -test statistics. Estimation algorithms induced by different matrix norms are developed for approximating the value of the unknown parameter. Modifications of existing multiple testing approaches are proposed by incorporating test dependence and enhancing their power through integration of Kolmogorov-Smirnov tests. Simulation studies are performed to demonstrate that the recommended methods preserve FDR control while improving power compared to traditional techniques.

Large-scale multiple testing with correlated tests and auxiliary statistics arises in a wide range of scientific fields. Conventional multiple testing procedures largely ignored auxiliary information, such as sparsity information, and the dependence structure among tests. This may result in loss of testing efficiency. In this paper, we propose a procedure, called multivariate local index of significance (mvLIS) procedure, for large-scale multiple testing. The mvLIS procedure can not only characterize local correlations among tests via a Markov chain but also incorporates auxiliary information via multivariate statistics. We present that the oracle mvLIS procedure is valid, namely, it controls false discovery rate (FDR) at the pre-specified level, and show that it yields the smallest false non-discovery rate (FNR) at the same FDR level. Then a data-driven mvLIS procedure is developed to mimic the oracle procedure. Comprehensive simulation studies and a real data analysis of schizophrenia (SCZ) data are performed to illustrate the superior performance of the mvLIS procedure. Moreover, as a byproduct that is of independent interest, we generalize the single-index modulated (SIM) multiple testing procedure, which embeds prior information via 2-dimensional p-values, to allow for d-dimensional ( d ≥ 3 ) statistics in multiple testing. The detailed extension is deferred to Discussion.

When a large number of hypotheses are investigated the false discovery rate (FDR) is commonly applied in gene expression analysis or gene association studies. Conventional single-stage designs may lack power due to low sample sizes for the individual hypotheses. We propose two-stage designs where the first stage is used to screen the 'promising' hypotheses which are further investigated at the second stage with an increased sample size. A multiple test procedure based on sequential individual P-values is proposed to control the FDR for the case of independent normal distributions with known variance. The power of optimal two-stage designs is impressively larger than the power of the corresponding single-stage design with equal costs. Extensions to the case of unknown variances and correlated test statistics are investigated by simulations. Moreover, it is shown that the simple multiple test procedure using first stage data for screening purposes and deriving the test decisions only from second stage data is a very powerful option.

This study introduces a novel p-value-based multiple testing approach tailored for generalized linear models. Despite the crucial role of generalized linear models in statistics, existing methodologies face obstacles arising from the heterogeneous variance of response variables and complex dependencies among estimated parameters. Our aim is to address the challenge of controlling the false discovery rate (FDR) amidst arbitrarily dependent test statistics. Through the development of efficient computational algorithms, we present a versatile statistical framework for multiple testing. The proposed framework accommodates a range of tools developed for constructing a new model matrix in regression-type analysis, including random row permutations and Model-X knockoffs. We devise efficient computing techniques to solve the encountered non-trivial quadratic matrix equations, enabling the construction of paired p-values suitable for the two-step multiple testing procedure proposed by Sarkar and Tang (Biometrika 109(4): 1149–1155, 2022). Theoretical analysis affirms the properties of our approach, demonstrating its capability to control the FDR at a given level. Empirical evaluations further substantiate its promising performance across diverse simulation settings.

ABSTRACTMaking accurate inference for gene regulatory networks, including inferring about pathway-by-pathway interactions, is an important and difficult task. Motivated by such genomic applications, we consider multiple testing for conditional dependence between subgroups of variables. Under a Gaussian graphical model framework, the problem is translated into simultaneous testing for a collection of submatrices of a high-dimensional precision matrix with each submatrix summarizing the dependence structure between two subgroups of variables.A novel multiple testing procedure is proposed and both theoretical and numerical properties of the procedure are investigated. Asymptotic null distribution of the test statistic for an individual hypothesis is established and the proposed multiple testing procedure is shown to asymptotically control the false discovery rate (FDR) and false discovery proportion (FDP) at the prespecified level under regularity conditions. Simulations show that the procedure works well in controlling the FDR and has good power in detecting the true interactions. The procedure is applied to a breast cancer gene expression study to identify between pathway interactions. Supplementary materials for this article are available online.

Mielke etal. (2021) proposed a stepwise multiple testing procedure (MTP) based on marginal p-values for rejecting at least k out of m null hypotheses. Briefly, the MTP cannot reject less than k hypotheses, but this property improves the power to reject k or more hypotheses relative to the stepdown MTP of Holm (1979). Mielke etal. discussed why such an MTP is of interest in the context of biosimilarity developments. This article describes how a slight modification of Holm's simple direct arguments can be used as an alternative to the closed-testing arguments of Mielke etal. to show the strong control of the family-wise error rate. This modification is based on a Holm-type formulation of the rejection algorithm where each step involves a single ordered p-value. With k equal to one, the stepwise MTP reduces to Holm's stepdown MTP. The MTP is valid quite generally. A version of the MTP with weights for hypotheses is also described and discussed. As in the case without weights: (a) a modification of Holm's arguments can be used; (b) with k equal to one, the MTP reduces to Holm's MTP with weights; and (c) the MTP is valid quite generally.

We consider large-scale studies in which thousands of significance tests are performed simultaneously. In some of these studies, the multiple testing procedure can be severely biased by latent confounding factors such as batch effects and unmeasured covariates that correlate with both primary variable(s) of interest (e.g., treatment variable, phenotype) and the outcome. Over the past decade, many statistical methods have been proposed to adjust for the confounders in hypothesis testing. We unify these methods in the same framework, generalize them to include multiple primary variables and multiple nuisance variables, and analyze their statistical properties. In particular, we provide theoretical guarantees for RUV-4 [Gagnon-Bartsch, Jacob and Speed (2013)] and LEAPP [Ann. Appl. Stat. 6 (2012) 1664-1688], which correspond to two different identification conditions in the framework: the first requires a set of "negative controls" that are known a priori to follow the null distribution; the second requires the true nonnulls to be sparse. Two different estimators which are based on RUV-4 and LEAPP are then applied to these two scenarios. We show that if the confounding factors are strong, the resulting estimators can be asymptotically as powerful as the oracle estimator which observes the latent confounding factors. For hypothesis testing, we show the asymptotic z-tests based on the estimators can control the type I error. Numerical experiments show that the false discovery rate is also controlled by the Benjamini-Hochberg procedure when the sample size is reasonably large.

Identifying influential variables in high-dimensional matrix-valued data while controlling the false discovery rate (FDR) is a critical challenge in modern data science. We propose a novel, model-free procedure specifically designed for simultaneous row and column selection in matrix predictor regression. Our approach utilizes folding selection subspaces (FSS) to formulate structured hypotheses and employs data splitting to construct mirror statistics from FSS estimators. This design bypasses restrictive model specification and the need for p-value computation. Key theoretical contributions include establishing the asymptotic distribution of FSS estimators and proving the mirror statistic is asymptotically symmetric with respect to zero under the null hypothesis. Using the symmetry, we develop a multiple hypothesis testing procedure with data-driven thresholds that provably controls the FDR for row and column at the desired level asymptotically. The framework is further extended to control element-wise FDR under specific structural assumptions. Extensive simulations and a real data analysis demonstrate the superior performance of the proposed method over existing approaches across various settings.

False Discovery Rate (FDR) and the Bayes risk are two different statistical measures, which can be used to evaluate and compare multiple testing procedures. Recent results show that under sparsity FDR controlling procedures, like the popular Benjamini-Hochberg (BH) procedure, perform also very well in terms of the Bayes risk. In particular asymptotic Bayes optimality under sparsity (ABOS) of BH was shown previously for location and scale models based on log-concave densities. This article extends previous work to a substantially larger set of distributions of effect sizes under the alternative, where the alternative distribution of true signals does not change with the number of tests $m$, while the sample size $n$ slowly increases. ABOS of BH and the corresponding step-down procedure based on FDR levels proportional to $n^{-1/2}$ are proved. A simulation study shows that these asymptotic results are relevant already for relatively small values of $m$ and $n$. Apart from showing asymptotic optimality of BH, our results on the optimal FDR level provide a natural extension of the well known results on the significance levels of Bayesian tests.

This paper provides an alternative approach to penalized regression for model selection in the context of high‐dimensional linear regressions where the number of covariates is large, often much larger than the number of available observations. We consider the statistical significance of individual covariates one at a time, while taking full account of the multiple testing nature of the inferential problem involved. We refer to the proposed method as One Covariate at a Time Multiple Testing (OCMT) procedure, and use ideas from the multiple testing literature to control the probability of selecting the approximating model, the false positive rate, and the false discovery rate. OCMT is easy to interpret, relates to classical statistical analysis, is valid under general assumptions, is faster to compute, and performs well in small samples. The usefulness of OCMT is also illustrated by an empirical application to forecasting U.S. output growth and inflation.

Multiple hypothesis testing is concerned with controlling the rate of false positives when testing several hypotheses simultaneously. One multiple hypothesis testing error measure is the false discovery rate (FDR), which is loosely defined to be the expected proportion of false positives among all significant hypotheses. The FDR is especially appropriate for exploratory analyses in which one is interested in finding several significant results among many tests. In this work, we introduce a modified version of the FDR called the “positive false discovery rate” (pFDR). We discuss the advantages and disadvantages of the pFDR and investigate its statistical properties. When assuming the test statistics follow a mixture distribution, we show that the pFDR can be written as a Bayesian posterior probability and can be connected to classification theory. These properties remain asymptotically true under fairly general conditions, even under certain forms of dependence. Also, a new quantity called the “q-value” is introduced and investigated, which is a natural “Bayesian posterior p-value,” or rather the pFDR analogue of the p-value. 1. Introduction. When testing a single hypothesis, one is usually concerned with controlling the false positive rate while maximizing the probability of detecting an effect when one really exists. In statistical terms, we maximize the power conditional on the Type I error rate being at or below some level. The field of multiple hypothesis testing tries to extend this basic paradigm to the situation where several hypotheses are tested simultaneously. One must define an appropriate compound error measure according to the rate of false positives one is willing to encounter. Then a procedure is developed that allows one to control the error rate at a desired level, while maintaining the power of each test as much as possible. The most commonly controlled quantity when testing multiple hypotheses is the family wise error rate (FWER), which is the probability of yielding one or more false positives out of all hypotheses tested. The most familiar example of this is the Bonferroni method. If there are m hypothesis tests, each test is controlled so that the probability of a false positive is less than or equal to α/m for some chosen value of α. It then follows that the overall FWER is less than or equal to α .M any

BackgroundFor gene expression or gene association studies with a large number of hypotheses the number of measurements per marker in a conventional single-stage design is often low due to limited resources. Two-stage designs have been proposed where in a first stage promising hypotheses are identified and further investigated in the second stage with larger sample sizes. For two types of two-stage designs proposed in the literature we derive multiple testing procedures controlling the False Discovery Rate (FDR) demonstrating FDR control by simulations: designs where a fixed number of top-ranked hypotheses are selected and designs where the selection in the interim analysis is based on an FDR threshold. In contrast to earlier approaches which use only the second-stage data in the hypothesis tests (pilot approach), the proposed testing procedures are based on the pooled data from both stages (integrated approach).ResultsFor both selection rules the multiple testing procedures control the FDR in the considered simulation scenarios. This holds for the case of independent observations across hypotheses as well as for certain correlation structures. Additionally, we show that in scenarios with small effect sizes the testing procedures based on the pooled data from both stages can give a considerable improvement in power compared to tests based on the second-stage data only.ConclusionThe proposed hypothesis tests provide a tool for FDR control for the considered two-stage designs. Comparing the integrated approaches for both selection rules with the corresponding pilot approaches showed an advantage of the integrated approach in many simulation scenarios.

Many species on Earth can carry up to two different versions of a given gene, with each of these ‘alleles’ having only a 50/50 chance of being transmitted to the next generation via sexual reproduction. Certain ‘selfish’ sequences, however, can hijack this process and increase their probability of being passed on to an offspring. Known as transmission distortion, this phenomenon may result in alleles spreading through the population even if they are detrimental to fertility. Transmission distortion has been detected in many species such as flies, mice and some plants. It can take place at various stages during reproduction; for example, the selfish alleles may become overrepresented among eggs or sperm. However, scientists need to study a large number of offspring or reproductive cells to be able to detect whether an allele is inherited more often than expected. This has made it difficult to determine whether transmission distortion also happens in humans, and research so far has resulted in conflicting conclusions. A recently published dataset of human sperm from 25 donors offered Carioscia, Weaver et al. the opportunity to examine this question. Every volunteer had produced between 969 and 3377 sperm cells, each with about 1% of their genome sequenced. Carioscia, Weaver et al. developed a computational method, which they named rhapsodi, that allowed them to ‘fill in the gaps’ and infer missing regions of the genome for each cell. To do so, they relied on the fact that sperm cells from a given individual are highly related to one another. With this more complete data at hand, it became possible to look for evidence of transmission distortion by searching for alleles that were overrepresented in sperm from a given donor. No selfish sequence could be detected in any of the 25 individuals, suggesting that human sperm may not be subject to pervasive transmission distortion. Signatures of selfish alleles detected in previous human studies may have therefore not resulted from this mechanism taking place at the sperm level. Instead, transmission distortion in humans could primarily target eggs or happen at later stages (for instance, if embryos carrying the selfish allele have better chances of survival). The ‘rhapsodi’ method developed by Carioscia, Weaver et al. should allow other scientists to work with datasets for which large portions of the genetic information is missing. It may therefore become easier for researchers to track selfish alleles which are difficult to detect, and to examine bigger, more diverse samples which also include individuals with known fertility challenges.