False discovery rate control with multivariate p-values

The multiplicity problem has become increasingly important in genetic studies as the capacity for high-throughput genotyping has increased. The control of False Discovery Rate (FDR) (Benjamini and Hochberg. [1995] J. R. Stat. Soc. Ser. B 57:289-300) has been adopted to address the problems of false positive control and low power inherent in high-volume genome-wide linkage and association studies. In many genetic studies, there is often a natural stratification of the m hypotheses to be tested. Given the FDR framework and the presence of such stratification, we investigate the performance of a stratified false discovery control approach (i.e. control or estimate FDR separately for each stratum) and compare it to the aggregated method (i.e. consider all hypotheses in a single stratum). Under the fixed rejection region framework (i.e. reject all hypotheses with unadjusted p-values less than a pre-specified level and then estimate FDR), we demonstrate that the aggregated FDR is a weighted average of the stratum-specific FDRs. Under the fixed FDR framework (i.e. reject as many hypotheses as possible and meanwhile control FDR at a pre-specified level), we specify a condition necessary for the expected total number of true positives under the stratified FDR method to be equal to or greater than that obtained from the aggregated FDR method. Application to a recent Genome-Wide Association (GWA) study by Maraganore et al. ([2005] Am. J. Hum. Genet. 77:685-693) illustrates the potential advantages of control or estimation of FDR by stratum. Our analyses also show that controlling FDR at a low rate, e.g. 5% or 10%, may not be feasible for some GWA studies.

The methodological advancement in microarray data analysis on the basis of false discovery rate (FDR) control, such as the q-value plots, allows the investigator to examine the FDR from several perspectives. However, when FDR control at the "customary" levels 0.01, 0.05, or 0.1 does not provide fruitful findings, there is little guidance for making the trade off between the significance threshold and the FDR level by sound statistical or biological considerations. Thus, meaningful statistical significance criteria that complement the existing FDR methods for large-scale multiple tests are desirable. Three statistical significance criteria, the profile information criterion, the total error proportion, and the guide-gene driven selection, are developed in this research. The first two are general significance threshold criteria for large-scale multiple tests; the profile information criterion is related to the recent theoretical studies of the connection between FDR control and minimax estimation, and the total error proportion is closely related to the asymptotic properties of FDR control in terms of the total error risk. The guide-gene driven selection is an approach to combining statistical significance and the existing biological knowledge of the study at hand. Error properties of these criteria are investigated theoretically and by simulation. The proposed methods are illustrated and compared using an example of genomic screening for novel Arf gene targets. Operating characteristics of q-value and the proposed significance threshold criteria are investigated and compared in a simulation study that employs a model mimicking a gene regulatory pathway. A guideline for using these criteria is provided. Splus/R code is available from the corresponding author upon request.

The recent RNA-seq technology is an attractive method to study gene expression. One of the most important goals in RNA-seq data analysis is to detect genes differentially expressed across treatments. Although several statistical methods have been published, there are no theoretical justifications for whether these methods are optimal or how to search for the optimal test. Furthermore, most proposed tests are designed for testing whether the mean expression levels are exactly the same or not across treatments, whereas sometimes, biologists are interested in detecting genes with expression changes larger than a certain threshold. Another issue with current methods is that the false discovery rate (FDR) control is not well studied. In this manuscript, we propose a test to address all the above issues. Under model assumptions, we derive an optimal test that achieves the maximum of average power among those that control FDR at the same level. We also provide an approximated version, the approximated most average powerful (AMAP) test, for practical implementation. The proposed method allows for testing null hypotheses that are much more general than the ones most previous studies have considered, and it leads to a natural way of controlling the FDR. Through simulation studies, we show that our test has a higher power than other methods, including the widely-used edgeR, DESeq, and baySeq methods, as well as better FDR control than two other FDR control procedures commonly used in practice. For demonstration, we also apply the proposed method to a real RNA-seq dataset obtained from maize.

False discovery rate (FDR) control serves as a critical safeguard in high-dimensional statistical analysis, increasing reliability and ensuring reproducibility of discoveries. Although extensively studied for real-valued data, FDR methodologies remain underdeveloped for the complex-valued data domain despite their importance in signal processing, physics, and engineering applications. This work confronts a central challenge in FDR-controlled analysis: settings in which the data contain groups of highly correlated variables. To achieve this, we advance our Complex-Valued Terminating-Random Experiments (CT-Rex) framework by developing the CT-Rex+GVS, a grouped variable selector. The method comes in two variants: the elastic net <tex>$(E N)$</tex> and the informed elastic net (IEN), both with isotropic and phase agnostic regularization. We validate the framework by benchmarking it in the selection of complex-valued grouped variables for linear regression and in the estimation of the singlesnapshot direction-of-arrival (DOA) based on compressed sensing using uniform linear arrays (ULA). The conducted numerical experiments confirm the FDR control property and demonstrate favorable performance compared to existing approaches.

Recent advances in long-read sequencing technologies have empowered the detection of structural variations (SVs) associated with genetic diseases. Despite the availability of numerous SV callers and efforts to merge SVs from multiple tools, there remains limited research on quantifying the confidence levels for the reported results. In this work, we propose a Bayesian integration model that combines SV calls from different tools. Notably, we introduce an approach for false discovery rate (FDR) control and provide a quantitative measure for the merged SVs. Our model can handle cases where certain tools lack quality scores, showcasing its flexibility in incorporating additional tools. Through extensive simulation studies, we evaluate the performance of our method under various conditions, demonstrating the FDR estimation accuracy and improved F1 score. Furthermore, we validate our model using simulated human genome sequencing data and the HG002dataset.

Multiple hypothesis testing is concerned with controlling the rate of false positives when testing several hypotheses simultaneously. One multiple hypothesis testing error measure is the false discovery rate (FDR), which is loosely defined to be the expected proportion of false positives among all significant hypotheses. The FDR is especially appropriate for exploratory analyses in which one is interested in finding several significant results among many tests. In this work, we introduce a modified version of the FDR called the “positive false discovery rate” (pFDR). We discuss the advantages and disadvantages of the pFDR and investigate its statistical properties. When assuming the test statistics follow a mixture distribution, we show that the pFDR can be written as a Bayesian posterior probability and can be connected to classification theory. These properties remain asymptotically true under fairly general conditions, even under certain forms of dependence. Also, a new quantity called the “q-value” is introduced and investigated, which is a natural “Bayesian posterior p-value,” or rather the pFDR analogue of the p-value. 1. Introduction. When testing a single hypothesis, one is usually concerned with controlling the false positive rate while maximizing the probability of detecting an effect when one really exists. In statistical terms, we maximize the power conditional on the Type I error rate being at or below some level. The field of multiple hypothesis testing tries to extend this basic paradigm to the situation where several hypotheses are tested simultaneously. One must define an appropriate compound error measure according to the rate of false positives one is willing to encounter. Then a procedure is developed that allows one to control the error rate at a desired level, while maintaining the power of each test as much as possible. The most commonly controlled quantity when testing multiple hypotheses is the family wise error rate (FWER), which is the probability of yielding one or more false positives out of all hypotheses tested. The most familiar example of this is the Bonferroni method. If there are m hypothesis tests, each test is controlled so that the probability of a false positive is less than or equal to α/m for some chosen value of α. It then follows that the overall FWER is less than or equal to α .M any

BackgroundFor gene expression or gene association studies with a large number of hypotheses the number of measurements per marker in a conventional single-stage design is often low due to limited resources. Two-stage designs have been proposed where in a first stage promising hypotheses are identified and further investigated in the second stage with larger sample sizes. For two types of two-stage designs proposed in the literature we derive multiple testing procedures controlling the False Discovery Rate (FDR) demonstrating FDR control by simulations: designs where a fixed number of top-ranked hypotheses are selected and designs where the selection in the interim analysis is based on an FDR threshold. In contrast to earlier approaches which use only the second-stage data in the hypothesis tests (pilot approach), the proposed testing procedures are based on the pooled data from both stages (integrated approach).ResultsFor both selection rules the multiple testing procedures control the FDR in the considered simulation scenarios. This holds for the case of independent observations across hypotheses as well as for certain correlation structures. Additionally, we show that in scenarios with small effect sizes the testing procedures based on the pooled data from both stages can give a considerable improvement in power compared to tests based on the second-stage data only.ConclusionThe proposed hypothesis tests provide a tool for FDR control for the considered two-stage designs. Comparing the integrated approaches for both selection rules with the corresponding pilot approaches showed an advantage of the integrated approach in many simulation scenarios.

BackgroundThe Significance Analysis of Microarrays (SAM) is a popular method for detecting significantly expressed genes and controlling the false discovery rate (FDR). Recently, it has been reported in the literature that the FDR is not well controlled by SAM. Due to the vast application of SAM in microarray data analysis, it is of great importance to have an extensive evaluation of SAM and its associated R-package (sam2.20).ResultsOur study has identified several discrepancies between SAM and sam2.20. One major difference is that SAM and sam2.20 use different methods for estimating FDR. Such discrepancies may cause confusion among the researchers who are using SAM or are developing the SAM-like methods. We have also shown that SAM provides no meaningful estimates of FDR and this problem has been corrected in sam2.20 by using a different formula for estimating FDR. However, we have found that, even with the improvement sam2.20 has made over SAM, sam2.20 may still produce erroneous and even conflicting results under certain situations. Using an example, we show that the problem of sam2.20 is caused by its use of asymmetric cutoffs which are due to the large variability of null scores at both ends of the order statistics. An obvious approach without the complication of the order statistics is the conventional symmetric cutoff method. For this reason, we have carried out extensive simulations to compare the performance of sam2.20 and the symmetric cutoff method. Finally, a simple modification is proposed to improve the FDR estimation of sam2.20 and the symmetric cutoff method.ConclusionOur study shows that the most serious drawback of SAM is its poor estimation of FDR. Although this drawback has been corrected in sam2.20, the control of FDR by sam2.20 is still not satisfactory. The comparison between sam2.20 and the symmetric cutoff method reveals that the relative performance of sam2.20 to the symmetric cutff method depends on the ratio of induced to repressed genes in a microarray data, and is also affected by the ratio of DE to EE genes and the distributions of induced and repressed genes. Numerical simulations show that the symmetric cutoff method has the biggest advantage over sam2.20 when there are equal number of induced and repressed genes (i.e., the ratio of induced to repressed genes is 1). As the ratio of induced to repressed genes moves away from 1, the advantage of the symmetric cutoff method to sam2.20 is gradually diminishing until eventually sam2.20 becomes significantly better than the symmetric cutoff method when the differentially expressed (DE) genes are either all induced or all repressed genes. Simulation results also show that our proposed simple modification provides improved control of FDR for both sam2.20 and the symmetric cutoff method.

Multi-testing correction methods are usually applied in more than one single nucleotide polymorphisms (SNPs) of association analyses to determine the significance of results in the face of testing multiple hypotheses. One of the statistical methods applied in these analyses is called false discovery rate (FDR) procedure. In this study, we compared the FDR theories behind the statistical software of Statistical Analysis Software (SAS) and Golden Helix using a database obtained from a methadone maintenance treatment cohort in Taiwan. The association analyses were performed between the delta opioid receptor (OPRD1) of seven SNPs from rs2236861 to rs760588, or four SNPs from rs2236861 to rs419335, and the diastolic blood pressure (DBP) in both genotypes and allele types. In general linear model (GLM) of association analyses, SNP rs797397 had the most significant association with DBP among seven or four SNPs of both genotype (p=0.0012) and allele type (p=0.0004) using SAS and Golden Helix programs. The genotype and allele type of this SNP had 0.0039 and 0.0018 FDR, respectively after Benjamini and Hochberg (BH) procedure analyses using SAS program, and had 0.0083 and 0.0027 FDR, respectively after the positive false discovery rate (pFDR) analyses using Golden Helix program in seven SNPs. The genotype and allele type of this SNP had 0.0022 and 0.0015 FDR, respectively after BH analyses and 0.0047 and 0.0015 FDR, respectively after pFDR analyses in four SNPs. In summary, the pFDR had less significant level than BH procedure in the most significant GLM associations SNP in both genotypes and allele types at seven and four numbers of SNPs. The FDR values were the same in both BH and pFDR multiple corrections in the allele type of four SNPs. The BH procedure in SAS provided a more consistent FDR in both genotype and allele type of multiple correction analyses despite the number of SNPs.

A topic which is becoming more and more popular in Functional Data Analysis is local inference, i.e., the continuous statistical testing of a null hypothesis along a domain of interest. The principal issue in this topic is the infinite amount of tested hypotheses, which can be seen as an extreme case of multiple comparisons problem.Anumber of quantities have been introduced in the literature of multivariate analysis in relation to the multiple comparisons problem. Arguably the most popular one is the False Discovery Rate (FDR), that measures the expected proportion of false discoveries (rejected null hypotheses) among all discoveries.We define FDR in the setting of functional data defined on a compact set of Rd. A continuous version of the Benjamini-Hochberg procedure is introduced, along with a definition of adjusted 𝑝-value function. Some general conditions are stated, under which the functional Benjamini-Hochberg (fBH) procedure provides control of FDR. We show how the procedure can be plugged-in with every parametric or nonparametric pointwise test, given that such test is exact. Finally, the proposed method - together with a nonparametric test - is applied to the analysis of the benchmark dataset of Canadian temperatures.

ABSTRACTIn many scientific fields, it is interesting and important to determine whether an observed data stream comes from a prespecified model or not, particularly when the number of data streams is of large scale, where multiple hypotheses testing is necessary. In this article, we consider large-scale model checking under certain dependence among different data streams observed at the same time. We propose a false discovery rate (FDR) control procedure to check those unusual data streams. Specifically, we derive an approximation of false discovery and construct a point estimate of FDR. Theoretical results show that, under some mild assumptions, our proposed estimate of FDR is simultaneously conservatively consistent with the true FDR, and hence it is an asymptotically strong control procedure. Simulation comparisons with some competing procedures show that our proposed FDR procedure behaves better in general settings. Application of our proposed FDR procedure is illustrated by the StarPlus fMRI data.

Testing multiple null hypotheses in two stages to decide which of these can be rejected or accepted at the first stage and which should be followed up for further testing having had additional observations is of importance in many scientific studies. We develop two procedures, each with two different combination functions, Fisher's and Simes’, to combine p-values from two stages, given prespecified boundaries on the first-stage p-values in terms of the false discovery rate (FDR) and controlling the overall FDR at a desired level. The FDR control is proved when the pairs of first- and second-stage p-values are independent and those corresponding to the null hypotheses are identically distributed as a pair (p 1, p 2) satisfying the p-clud property. We did simulations to show that (1) our two-stage procedures can have significant power improvements over the first-stage Benjamini–Hochberg (BH) procedure compared to the improvement offered by the ideal BH procedure that one would have used had the second stage data been available for all the hypotheses, and can continue to control the FDR under some dependence situations, and (2) can offer considerable cost savings compared to the ideal BH procedure. The procedures are illustrated through a real gene expression data. Supplementary materials for this article are available online.

Comparing the spatial characteristics of spatiotemporal random fields is often at demand. However, the comparison can be challenging due to the high-dimensional feature and dependency in the data. We develop a new multiple testing approach to detect local differences in the spatial characteristics of two spatiotemporal random fields by taking the spatial information into account. Our method adopts a two-component mixture model for location wise p-values and then derives a new false discovery rate (FDR) control, called mirror procedure, to determine the optimal rejection region. This procedure is robust to model misspecification and allows for weak dependency among hypotheses. To integrate the spatial heterogeneity, we model the mixture probability as well as study the benefit if any of allowing the alternative distribution to be spatially varying. An EM-algorithm is developed to estimate the mixture model and implement the FDR procedure. We study the FDR control and the power of our new approach both theoretically and numerically, and apply the approach to compare the mean and teleconnection pattern between two synthetic climate fields. Supplementary materials for this article are available online.

High-throughput microarray technology can be used to examine thousands of features, such as all the genes of an organism, and measure their expression. Two important issues of microarray bioinformatics are first, how to combine the significance values for each feature across experiments with high statistical power, and second, how to control the proportion of false positives. Existing methods address these issues separately, in spite of their linked usage. We present a novel method (ESP) to address the two requirements in an interdependent way. It generalizes the truncated product method of Zaykin et al. to combine only those significance values which clear their respective experiment-specific false discovery restrictive thresholds, thus allowing us to control the false discovery rate (FDR) for the final combined result. Further, we introduce several concepts that together offer FDR control, high power, quality control and speed-up in meta-analysis as done by our algorithm. Computational and statistical methods of research synthesis like the one described here will be increasingly important as additional genome-wide datasets accumulate in databases. We apply our method to combine three well-known ChIP-chip transcription factor binding datasets for budding yeast to identify significant intergenic regulatory sequences for nine cell cycle regulating transcription factors, both with high power and controlled FDR.

Quick detection of common changes is critical in sequential monitoring of multi-stream data where a common change is referred as a change that only occurs in a portion of panels. After a common change is detected by using a SR-SUM procedure, we compare the CUSUM and EWMA methods in terms of FDR and FNR after a common change is detected. The BH-type procedure is used for both CUSUM processes and EWMA processes at the detection time to isolate the changed panels with the control of FDR (False Discovery Rate). The numerical results show that the FNR for the CUSUM method is smaller than that for the EWMA method. Also the CUSUM method has the advantage that the common change point is conveniently estimated after isolation. To reduce the FNR, the tightened FDR control procedure is considered by using the estimated number of changed panels. Further, the estimated number of changed panels can also be used to identify whether the alarm is true or false.