Multiple hypothesis testing in proteomics: A strategy for experimental work.

Estimating financial market volatility is integral to the study of investment decisions and behaviour. Previous literature has, therefore, attempted to identify an optimal volatility forecasting model. However, optimal volatility forecasting is dynamic. It depends on the asset being studied and financial market conditions. We propose a novel empirical methodology to account for this dynamism. Using our Multiple Hypothesis Testing with the False Discovery Rate (FDR) method, we identify buckets of superior-performing models relative to the literature’s benchmark models. We present evidence that our proposed FDR bucket with GJR-GARCH has the lowest forecast error in predicting one-step-ahead realized volatility. We also compare our FDR method with two Family-Wise Error Rate model selection frameworks, and the evidence supports our proposed FDR methodology.

Multiple hypothesis testing is concerned with controlling the rate of false positives when testing several hypotheses simultaneously. One multiple hypothesis testing error measure is the false discovery rate (FDR), which is loosely defined to be the expected proportion of false positives among all significant hypotheses. The FDR is especially appropriate for exploratory analyses in which one is interested in finding several significant results among many tests. In this work, we introduce a modified version of the FDR called the “positive false discovery rate” (pFDR). We discuss the advantages and disadvantages of the pFDR and investigate its statistical properties. When assuming the test statistics follow a mixture distribution, we show that the pFDR can be written as a Bayesian posterior probability and can be connected to classification theory. These properties remain asymptotically true under fairly general conditions, even under certain forms of dependence. Also, a new quantity called the “q-value” is introduced and investigated, which is a natural “Bayesian posterior p-value,” or rather the pFDR analogue of the p-value. 1. Introduction. When testing a single hypothesis, one is usually concerned with controlling the false positive rate while maximizing the probability of detecting an effect when one really exists. In statistical terms, we maximize the power conditional on the Type I error rate being at or below some level. The field of multiple hypothesis testing tries to extend this basic paradigm to the situation where several hypotheses are tested simultaneously. One must define an appropriate compound error measure according to the rate of false positives one is willing to encounter. Then a procedure is developed that allows one to control the error rate at a desired level, while maintaining the power of each test as much as possible. The most commonly controlled quantity when testing multiple hypotheses is the family wise error rate (FWER), which is the probability of yielding one or more false positives out of all hypotheses tested. The most familiar example of this is the Bonferroni method. If there are m hypothesis tests, each test is controlled so that the probability of a false positive is less than or equal to α/m for some chosen value of α. It then follows that the overall FWER is less than or equal to α .M any

It is a typical feature of high dimensional data analysis, for example a microarray study, that a researcher allows thousands of statistical tests at a time. All inferences for the tests are determined using the p-values; a smaller p-value than the α-level of the test signifies a statistically significant test. As the number of tests increases, the chance of observing some small p-values is very high even when all null hypotheses are true. Consequently, we make wrong conclusions on the hypotheses. This type of potential problem frequently happens when we test several hypotheses simultaneously, i.e., the multiple testing problem. Adjustment of the p-values can redress the problem that arises in multiple hypothesis testing. P-value adjustment methods control error rates [type I error (i.e. false positive) and type II error (i.e. false negative)] for each hypothesis in order to achieve high statistical power while keeping the overall Family Wise Error Rate (FWER) no larger than α, where α is most often set to 0.05. However, researchers also consider the False Discovery Rate (FDR), or Positive False Discovery Rate (pFDR) instead of the type I error in multiple comparison problems for microarray studies. The methods involved in controlling the FDR always provide higher statistical power than the methods involved in controlling the type I error rate while keeping the type II error rate low. In practice, microarray studies involve dependent test statistics (or p-values) because genes can be fully dependent on each other in a complicated biological structure. However, some of the p-value adjustment methods only deal with independent test statistics. Thus, we carry out a simulation study with several methods involved in multiple hypothesis testing.

Problems with analyzing and processing high-dimensional random vectors arise in a wide variety of areas. Important practical tasks are economical representation, searching for significant features, and removal of insignificant (noise) features. These tasks are fundamentally important for a wide class of practical applications, such as genetic chain analysis, encephalography, spectrography, video and audio processing, and a number of others. Current research in this area includes a wide range of papers devoted to various filtering methods based on the sparse representation of the obtained experimental data and statistical procedures for their processing. One of the most popular approaches to constructing statistical estimates of regularities in experimental data is the procedure of multiple testing of hypotheses about the significance of observations. In this paper, we consider a procedure based on the false discovery rate (FDR) measure that controls the expected percentage of false rejections of the null hypothesis. We analyze the asymptotic properties of the mean-square error estimate for this procedure and prove the statements about the asymptotic normality of this estimate. The obtained results make it possible to construct asymptotic confidence intervals for the mean-square error of the FDR method using only the observed data.

ABSTRACTIn many scientific fields, it is interesting and important to determine whether an observed data stream comes from a prespecified model or not, particularly when the number of data streams is of large scale, where multiple hypotheses testing is necessary. In this article, we consider large-scale model checking under certain dependence among different data streams observed at the same time. We propose a false discovery rate (FDR) control procedure to check those unusual data streams. Specifically, we derive an approximation of false discovery and construct a point estimate of FDR. Theoretical results show that, under some mild assumptions, our proposed estimate of FDR is simultaneously conservatively consistent with the true FDR, and hence it is an asymptotically strong control procedure. Simulation comparisons with some competing procedures show that our proposed FDR procedure behaves better in general settings. Application of our proposed FDR procedure is illustrated by the StarPlus fMRI data.

We propose a Bayesian method for multiple hypothesis testing in random effects models that uses Dirichlet process (DP) priors for a nonparametric treatment of the random effects distribution. We consider a general model formulation which accommodates a variety of multiple treatment conditions. A key feature of our method is the use of a product of spiked distributions, i.e., mixtures of a point-mass and continuous distributions, as the centering distribution for the DP prior. Adopting these spiked centering priors readily accommodates sharp null hypotheses and allows for the estimation of the posterior probabilities of such hypotheses. Dirichlet process mixture models naturally borrow information across objects through model-based clustering while inference on single hypotheses averages over clustering uncertainty. We demonstrate via a simulation study that our method yields increased sensitivity in multiple hypothesis testing and produces a lower proportion of false discoveries than other competitive methods. While our modeling framework is general, here we present an application in the context of gene expression from microarray experiments. In our application, the modeling framework allows simultaneous inference on the parameters governing differential expression and inference on the clustering of genes. We use experimental data on the transcriptional response to oxidative stress in mouse heart muscle and compare the results from our procedure with existing nonparametric Bayesian methods that provide only a ranking of the genes by their evidence for differential expression.

Testing multiple null hypotheses in two stages to decide which of these can be rejected or accepted at the first stage and which should be followed up for further testing having had additional observations is of importance in many scientific studies. We develop two procedures, each with two different combination functions, Fisher's and Simes’, to combine p-values from two stages, given prespecified boundaries on the first-stage p-values in terms of the false discovery rate (FDR) and controlling the overall FDR at a desired level. The FDR control is proved when the pairs of first- and second-stage p-values are independent and those corresponding to the null hypotheses are identically distributed as a pair (p 1, p 2) satisfying the p-clud property. We did simulations to show that (1) our two-stage procedures can have significant power improvements over the first-stage Benjamini–Hochberg (BH) procedure compared to the improvement offered by the ideal BH procedure that one would have used had the second stage data been available for all the hypotheses, and can continue to control the FDR under some dependence situations, and (2) can offer considerable cost savings compared to the ideal BH procedure. The procedures are illustrated through a real gene expression data. Supplementary materials for this article are available online.

In this paper we propose a computationally efficient multiple hypothesis testing procedure for persistent homology. The computational efficiency of our procedure is based on the observation that one can empirically simulate a null distribution that is universal across many hypothesis testing applications involving persistence homology. Our observation suggests that one can simulate the null distribution efficiently based on a small number of summaries of the collected data and use this null in the same way that p-value tables were used in classical statistics. To illustrate the efficiency and utility of the null distribution we provide procedures for rejecting acyclicity with both control of the Family-Wise Error Rate (FWER) and the False Discovery Rate (FDR). We will argue that the empirical null we propose is very general conditional on a few summaries of the data based on simulations and limit theorems for persistent homology for point processes.

False discovery rate (FDR) controlling procedures provide important statistical guarantees for replicability in signal identification based on multiple hypotheses testing. In many fields of study, FDR controling procedures are used in high-dimensional (HD) analyses to discover features that are truly associated with the outcome. In some recent applications, data on the same set of candidate features are independently collected in multiple different studies. For example, gene expression data are collected at different facilities and with different cohorts, to identify the genetic biomarkers of multiple types of cancers. These studies provide us with opportunities to identify signals by considering information from different sources (with potential heterogeneity) jointly. This paper is about how to provide FDR control guarantees for the tests of union null hypotheses of conditional independence. We present a knockoff-based variable selection method (Simultaneous knockoffs) to identify mutual signals from multiple independent datasets, providing exact FDR control guarantees under finite sample settings. This method can work with very general model settings and test statistics. We demonstrate the performance of this method with extensive numerical studies and two real-dataexamples.

Duality between multiple testing and selecting

The multiplicity problem has become increasingly important in genetic studies as the capacity for high-throughput genotyping has increased. The control of False Discovery Rate (FDR) (Benjamini and Hochberg. [1995] J. R. Stat. Soc. Ser. B 57:289-300) has been adopted to address the problems of false positive control and low power inherent in high-volume genome-wide linkage and association studies. In many genetic studies, there is often a natural stratification of the m hypotheses to be tested. Given the FDR framework and the presence of such stratification, we investigate the performance of a stratified false discovery control approach (i.e. control or estimate FDR separately for each stratum) and compare it to the aggregated method (i.e. consider all hypotheses in a single stratum). Under the fixed rejection region framework (i.e. reject all hypotheses with unadjusted p-values less than a pre-specified level and then estimate FDR), we demonstrate that the aggregated FDR is a weighted average of the stratum-specific FDRs. Under the fixed FDR framework (i.e. reject as many hypotheses as possible and meanwhile control FDR at a pre-specified level), we specify a condition necessary for the expected total number of true positives under the stratified FDR method to be equal to or greater than that obtained from the aggregated FDR method. Application to a recent Genome-Wide Association (GWA) study by Maraganore et al. ([2005] Am. J. Hum. Genet. 77:685-693) illustrates the potential advantages of control or estimation of FDR by stratum. Our analyses also show that controlling FDR at a low rate, e.g. 5% or 10%, may not be feasible for some GWA studies.

Aims: CD36 , an important gene involved in cholesterol and fatty acid metabolism,, has been associated with many metabolic-related traits, including diabetes, hypertension, and obesity in different populations. However, the association of this gene with metabolic syndrome (MS) has not been well-studied. We aimed to investigate the association of CD36 with the risk of MS and longitudinal changes of its components in among participants of the Rural Chinese Cohort Study (RCCS). Materials and Methods: The RCCS enrolled 20,194 participants aged 2 18 years old between July of 2007 and August of 2008. A total of 17,265 participants (response rate 85.5%) were examined in the follow-up survey between July of 2013 and October of 2014. During the six years, 729 new incident cases of MS were ascertained. We conducted a nested case-control study within the RCCS. Specifically, 408 MS randomly cases and 408 one-to-one matched controls were included in the current study. In addition to follow-up time, non-MS controls were matched to MS cases on age, gender, marital status, and residence village. Human genomic DNA was extracted using a standard method from blood samples collected at the baseline. After quality control, 4 single nucleotide polymorphisms (SNPs) within the CD36 were successfully genotyped. MS was defined according to the 2009 criteria proposed by IDF and AHA/NHLBI. Multivariate conditional logistic regression was used to determine additive associations of SNPs and SNP-haplotype with MS adjusting for body mass index (BMI), smoking, drinking, and physical activity at baseline. Multivariate linear regression models were conducted among the control group to examine association of SNPs and SNP-haplotypes with MS components, controlling for age, gender, BMI, smoking, drinking and physical activity. Multiple testing was corrected by false positive discovery rate (FDR) method. Results: Two SNPs, rs7755 and rs1049673, were associated with risk for MS. Per minor allele increase in rs7755 and rs1049673 was associated with 2.29 (95% confidence interval [CI]: 1.24-4.24, p =0.008) and 2.30 (95% CI: 1.24-4.26, p =0.008) times higher odds of MS, respectively. In addition, rs1194197 was associated with changes of triglycerides level (β=0.151, p =0.014), and rs7755 (β=-0.065, p =0.028) and rs1049673 (β=-0.067, p =0.025) were associated with changes of high density lipoprotein cholesterol in the control group. One haplotype of the four SNPs, AGAG, was associated with the change of waist circumference (β=-1.773, p =0.048). Conclusions: We identified important SNPs and SNP-haplotypes of the CD36 associated with MS and changes of MS components in a Han Chinese population.

Introduction Racial disparities in breast cancer (BrCa) mortality are well documented. We previously reported that African American (AA) women with estrogen receptor-positive (ER+) BrCa have a 4-fold higher rate of death compared to non-Hispanic whites (white) after controlling for stage at diagnosis and treatment initiation. This suggests that racial differences in tumor biology may contribute to the survival disparity. In preliminary work we found that AA BrCa cases have higher serum levels of C-reactive protein (CRP) compared to whites, and that increased CRP was associated with worse survival, suggesting that increased systemic inflammation in AA women drives aggressive tumor biology and contributes to the racial disparity in survival. The goal of this study is to identify specific inflammatory pathways that may be involved. Experimental Procedures Pre-treatment blood samples from women with stage I-III ER+ BrCa (AA, n=40; white, n=55) and from healthy controls (AA, n=51; white, n=38) were analyzed with the Olink® targeted proteomic assay, which measures 92 inflammatory proteins. Normalized data was log2 transformed, and generalized linear models compared the normalized intensities with covariates of interest. The fit of the model was tested with empirical Bayes methods. All p-values are corrected for multiple testing with the false discovery rate (FDR) method of Benjamini and Hochberg. Findings After adjustment for site of enrollment, plasma levels of 13 inflammatory cytokines showed a log2 fold-change (log2FC) ≥ 1.0 for AA compared to white women, i.e. at least a two-fold increase in the mean value on a linear scale (Table). The racial difference was evident among both cases and controls, and remained after adjusting for body mass index (BMI). There were no differences between AA cases and controls, indicating that cytokine elevation in AA BrCa patients is not an effect of the tumor. Several of the cytokines are known to promote tumor progression and aggressive biological features in breast tumors. ProteinLog2FCProteinLog2FCCXCL52.42**CCL81.13***CXCL12.06***EIF4EBP11.12**STAMBP1.91*CCL201.10**CXCL111.89**TNF141.06*CCL31.45**TGF beta1.04**CXCL61.40**CCL71.03***CCL131.16**FDR corrected p-values: * < 0.05, **< 0.01, ***< 0.001 Conclusion Systemic elevation of biologically relevant inflammatory cytokines in AA women with ER+ BrCa does not appear to be an effect of the tumor. This inflammatory phenotype may activate inflammatory programs in ER+ breast tumors that develop in AA women, leading to disproportionately aggressive tumor biology and worse outcomes. Supported by grants U54CA203000; U54CA2022995; and U54CA2022997 from the National Institutes of Health. Citation Format: Oana C. Danciu, George Chlipala, Zeynep Madak-Erdogan, Hariyali Patel, Landan Banks, Ayesha Zaidi, Ashlie Santaliz-Casiano, Lauren Schulte, Lauren Weller, Deanna Taiym, Natalie Pulliam, Elona Liko Hazizi, Elonia Martin, Anita Fareeduddin, Archana Bargaje, Carlos Garcia, Scott Hegerty, Sarah Friedewald, Seema Khan, J Julie Kim, William Gradishar, Garth Rauscher, Jonna Frasor, Kent F. Hoskins. Racial differences in circulating inflammatory cytokines and breast cancer disparities [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 1176.

Step-up and step-down procedures controlling the number and proportion of false positives

In this paper, we present a novel object-based statistical colocalization method. Our colocalization relies on multiple hypothesis tests on the distances between all pairs of the (spot-shaped) objects from the two markers. We wish to test among all these pairs how many are significantly close to each other such that they cannot occur just "by chance". Two objects are decided to be co-localized if the test on their distance is significant. For this purpose, we first extract the objects by applying a wavelet-based spot detection approach which fully takes into account the mixed-Poisson-Gaussian noise process of confocal fluorescence images. Then, we build a null hypothesis model in which the distribution of the distance between two independently randomly drawn detections in the cell is estimated by a kernel method. The observed distances are tested against this null model. Our tests control the false discovery rate (FDR) of the co-localizations. Simulations show that this approach has a good specificity. Furthermore, our method has been successfully applied in a real problem of protein colocalization analysis during the endocytic process.