A new p-value based multiple testing procedure for generalized linear models

Large-scale multiple testing with correlated tests and auxiliary statistics arises in a wide range of scientific fields. Conventional multiple testing procedures largely ignored auxiliary information, such as sparsity information, and the dependence structure among tests. This may result in loss of testing efficiency. In this paper, we propose a procedure, called multivariate local index of significance (mvLIS) procedure, for large-scale multiple testing. The mvLIS procedure can not only characterize local correlations among tests via a Markov chain but also incorporates auxiliary information via multivariate statistics. We present that the oracle mvLIS procedure is valid, namely, it controls false discovery rate (FDR) at the pre-specified level, and show that it yields the smallest false non-discovery rate (FNR) at the same FDR level. Then a data-driven mvLIS procedure is developed to mimic the oracle procedure. Comprehensive simulation studies and a real data analysis of schizophrenia (SCZ) data are performed to illustrate the superior performance of the mvLIS procedure. Moreover, as a byproduct that is of independent interest, we generalize the single-index modulated (SIM) multiple testing procedure, which embeds prior information via 2-dimensional p-values, to allow for d-dimensional ( d ≥ 3 ) statistics in multiple testing. The detailed extension is deferred to Discussion.

ABSTRACTMaking accurate inference for gene regulatory networks, including inferring about pathway-by-pathway interactions, is an important and difficult task. Motivated by such genomic applications, we consider multiple testing for conditional dependence between subgroups of variables. Under a Gaussian graphical model framework, the problem is translated into simultaneous testing for a collection of submatrices of a high-dimensional precision matrix with each submatrix summarizing the dependence structure between two subgroups of variables.A novel multiple testing procedure is proposed and both theoretical and numerical properties of the procedure are investigated. Asymptotic null distribution of the test statistic for an individual hypothesis is established and the proposed multiple testing procedure is shown to asymptotically control the false discovery rate (FDR) and false discovery proportion (FDP) at the prespecified level under regularity conditions. Simulations show that the procedure works well in controlling the FDR and has good power in detecting the true interactions. The procedure is applied to a breast cancer gene expression study to identify between pathway interactions. Supplementary materials for this article are available online.

The classical Benjamini–Hochberg (B-H) method, widely used across various disciplines such as genetics, epidemiology, and social sciences, serves as an established procedure for controlling the false discovery rate (FDR) in multiple comparison scenarios. The B-H method assumes independence among tests, which often does not hold in large-scale dependent datasets. The Benjamini–Yekutieli (B-Y) adjustment controls the FDR under arbitrary dependence but is often very conservative and can lead to a reduction in statistical power. This paper investigates the performance of the B-H and B-Y procedures under specific positive block dependence structures. Two parametric forms of block dependence are considered to model the correlation among paired t -test statistics. Estimation algorithms induced by different matrix norms are developed for approximating the value of the unknown parameter. Modifications of existing multiple testing approaches are proposed by incorporating test dependence and enhancing their power through integration of Kolmogorov-Smirnov tests. Simulation studies are performed to demonstrate that the recommended methods preserve FDR control while improving power compared to traditional techniques.

The burgeoning field of genomics has revived interest in multiple testing procedures by raising new methodological and computational challenges. For example, microarray experiments generate large multiplicity problems in which thousands of hypotheses are tested simultaneously. Westfall and Young (1993) propose resampling-based p-value adjustment procedures which are highly relevant to microarray experiments. This article discusses dierent criteria for error control in resampling-based multiple testing, including (a) the family wise error rate of Westfall and Young (1993) and (b) the false discovery rate developed by Benjamini and Hochberg (1995), both from a frequentist viewpoint; and (c) the positive false discovery rate of Storey (2002a), which has a Bayesian motivation. We also introduce our recently developed fast algorithm for implementing the minP adjustment to control family-wise error rate. Adjusted p-values for dierent approaches are applied to gene expression data from two recently published microarray studies. The properties of these procedures for multiple testing are compared.

Much effort has been made to improve the famous step up procedure of Benjamini and Hochberg given by linear critical values $\frac{i\alpha}{n}$. It is pointed out by Gavrilov, Benjamini and Sarkar that step down multiple testing procedures based on the critical values $\beta_{i}=\frac{i\alpha}{n+1-i(1-\alpha)}$ still control the false discovery rate (FDR) at the upper bound $\alpha$ under basic independence assumptions. Since that result is no longer true for step up procedures and for step down procedures, if the p-values are dependent, a big discussion about the corresponding FDR starts in the literature. The present paper establishes finite sample formulas and bounds for the FDR and the expected number of false rejections for multiple testing procedures using critical values $\beta_{i}$ under martingale and reverse martingale dependence models. It is pointed out that martingale methods are natural tools for the treatment of local FDR estimators which are closely connected to the present coefficients $\beta_{i}$. The martingale approach also yields new results and further inside for the special basic independence model.

Identifying influential variables in high-dimensional matrix-valued data while controlling the false discovery rate (FDR) is a critical challenge in modern data science. We propose a novel, model-free procedure specifically designed for simultaneous row and column selection in matrix predictor regression. Our approach utilizes folding selection subspaces (FSS) to formulate structured hypotheses and employs data splitting to construct mirror statistics from FSS estimators. This design bypasses restrictive model specification and the need for p-value computation. Key theoretical contributions include establishing the asymptotic distribution of FSS estimators and proving the mirror statistic is asymptotically symmetric with respect to zero under the null hypothesis. Using the symmetry, we develop a multiple hypothesis testing procedure with data-driven thresholds that provably controls the FDR for row and column at the desired level asymptotically. The framework is further extended to control element-wise FDR under specific structural assumptions. Extensive simulations and a real data analysis demonstrate the superior performance of the proposed method over existing approaches across various settings.

We investigate several distribution-free dependence detection procedures, all based on a shuffling of the trials, from a statistical point of view. The mathematical justification of such procedures lies in the bootstrap principle and its approximation properties. In particular, we show that such a shuffling has mainly to be done on centered quantities-that is, quantities with zero mean under independence-to construct correct p-values, meaning that the corresponding tests control their false positive (FP) rate. Thanks to this study, we introduce a method, named permutation UE, which consists of a multiple testing procedure based on permutation of experimental trials and delayed coincidence count. Each involved single test of this procedure achieves the prescribed level, so that the corresponding multiple testing procedure controls the false discovery rate (FDR), and this with as few assumptions as possible on the underneath distribution, except independence and identical distribution across trials. The mathematical meaning of this assumption is discussed, and it is in particular argued that it does not mean what is commonly referred in neuroscience to as cross-trials stationarity. Some simulations show, moreover, that permutation UE outperforms the trial-shuffling of Pipa and Grün ( 2003 ) and the MTGAUE method of Tuleau-Malot etal. ( 2014 ) in terms of single levels and FDR, for a comparable amount of false negatives. Application to real data is also provided.

Multiple testing problems are complex in evaluating statistical evidence in pivotal clinical trials for regulatory applications. However, a common practice is to employ a general and rather simple multiple comparison procedure to handle the problems. Applying multiple comparison adjustments is to ensure proper control of type I error rates. However, in many practices, the emphasis of the type I error rate control often leads to a choice of a statistically valid multiple test procedure but the common sense is overlooked. The challenges begin with confusions in defining a relevant family of hypotheses for which the type I error rates need to be properly controlled. Multiple testing problems are in a wide variety, ranging from testing multiple doses and endpoints jointly, composite endpoint, non-inferiority and superiority, to studying time of onset of a treatment effect, and searching for minimum effective dose or a patient subgroup in which the treatment effect lies. To select a valid and sensible multiple test procedure, the first step should be to tailor the selection to the study questions and to the ultimate clinical decision tree. Then evaluation of statistical power performance should come in to play in the next step to fine tune the selected procedure.

Mielke etal. (2021) proposed a stepwise multiple testing procedure (MTP) based on marginal p-values for rejecting at least k out of m null hypotheses. Briefly, the MTP cannot reject less than k hypotheses, but this property improves the power to reject k or more hypotheses relative to the stepdown MTP of Holm (1979). Mielke etal. discussed why such an MTP is of interest in the context of biosimilarity developments. This article describes how a slight modification of Holm's simple direct arguments can be used as an alternative to the closed-testing arguments of Mielke etal. to show the strong control of the family-wise error rate. This modification is based on a Holm-type formulation of the rejection algorithm where each step involves a single ordered p-value. With k equal to one, the stepwise MTP reduces to Holm's stepdown MTP. The MTP is valid quite generally. A version of the MTP with weights for hypotheses is also described and discussed. As in the case without weights: (a) a modification of Holm's arguments can be used; (b) with k equal to one, the MTP reduces to Holm's MTP with weights; and (c) the MTP is valid quite generally.

In this paper we propose a computationally efficient multiple hypothesis testing procedure for persistent homology. The computational efficiency of our procedure is based on the observation that one can empirically simulate a null distribution that is universal across many hypothesis testing applications involving persistence homology. Our observation suggests that one can simulate the null distribution efficiently based on a small number of summaries of the collected data and use this null in the same way that p-value tables were used in classical statistics. To illustrate the efficiency and utility of the null distribution we provide procedures for rejecting acyclicity with both control of the Family-Wise Error Rate (FWER) and the False Discovery Rate (FDR). We will argue that the empirical null we propose is very general conditional on a few summaries of the data based on simulations and limit theorems for persistent homology for point processes.

This paper reviews global and multiple tests for the combination ofn hypotheses using the orderedp-values of then individual tests. In 1987, Rohmel and Streitberg presented a general method to construct global level α tests based on orderedp-values when there exists no prior knowledge regarding the joint distribution of the corresponding test statistics. In the case of independent test statistics, construction of global tests is available by means of recursive formulae presented by Bicher (1989), Kornatz (1994) and Finner and Roters (1994). Multiple test procedures can be developed by applying the closed test principle using these global tests as building blocks. Liu (1996) proposed representing closed tests by means of “critical matrices” which contain the critical values of the global tests. Within the framework of these theoretical concepts, well-known global tests and multiple test procedures are classified and the relationships between the different tests are characterised.

When a large number of hypotheses are investigated the false discovery rate (FDR) is commonly applied in gene expression analysis or gene association studies. Conventional single-stage designs may lack power due to low sample sizes for the individual hypotheses. We propose two-stage designs where the first stage is used to screen the 'promising' hypotheses which are further investigated at the second stage with an increased sample size. A multiple test procedure based on sequential individual P-values is proposed to control the FDR for the case of independent normal distributions with known variance. The power of optimal two-stage designs is impressively larger than the power of the corresponding single-stage design with equal costs. Extensions to the case of unknown variances and correlated test statistics are investigated by simulations. Moreover, it is shown that the simple multiple test procedure using first stage data for screening purposes and deriving the test decisions only from second stage data is a very powerful option.

Multiplicity of data and compounding errors is often overlooked in data analysis for applied business scenarios. Statistical theory around multiple testing provides a framework for describing appropriate error rates and offers methods to control them in order to protect against wrong conclusions. However, these multiple testing procedures are often misunderstood and underutilised in applied business problems. In this article, existing multiple testing methodologies are reviewed and summarised. Specific numeric examples are shown to illustrate the techniques and demonstrate the statistical power of each. Finally, three cases are given of business-related situations when multiple testing can be overlooked in data analysis.

This paper provides an alternative approach to penalized regression for model selection in the context of high‐dimensional linear regressions where the number of covariates is large, often much larger than the number of available observations. We consider the statistical significance of individual covariates one at a time, while taking full account of the multiple testing nature of the inferential problem involved. We refer to the proposed method as One Covariate at a Time Multiple Testing (OCMT) procedure, and use ideas from the multiple testing literature to control the probability of selecting the approximating model, the false positive rate, and the false discovery rate. OCMT is easy to interpret, relates to classical statistical analysis, is valid under general assumptions, is faster to compute, and performs well in small samples. The usefulness of OCMT is also illustrated by an empirical application to forecasting U.S. output growth and inflation.

Efforts to develop more efficient multiple hypothesis testing procedures for false discovery rate (FDR) control have focused on incorporating an estimate of the proportion of true null hypotheses (such procedures are called adaptive) or exploiting heterogeneity across tests via some optimal weighting scheme. This paper combines these approaches using a weighted adaptive multiple decision function (WAMDF) framework. Optimal weights for a flexible random effects model are derived and a WAMDF that controls the FDR for arbitrary weighting schemes when test statistics are independent under the null hypotheses is given. Asymptotic and numerical assessment reveals that, under weak dependence, the proposed WAMDFs provide more efficient FDR control even if optimal weights are misspecified. The robustness and flexibility of the proposed methodology facilitates the development of more efficient, yet practical, FDR procedures for heterogeneous data. To illustrate, two different weighted adaptive FDR methods for heterogeneous sample sizes are developed and applied to data.