Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research.

Chapter Seven - The debate over null hypothesis testing and wise use as the solution

This paper presents a review and critique of statistical null hypothesis testing in ecological studies in general, and wildlife studies in particular, and describes an alternative. Our review of Ecology and the Journal of Wildlife Management found the use of null hypothesis testing to be pervasive. The estimated number of P-values appearing within articles of Ecology exceeded 8,000 in 1991 and has exceeded 3,000 in each year since 1984, whereas the estimated number of P-values in the Journal of Wildlife Management exceeded 8,000 in 1997 and has exceeded 3,000 in each year since 1994. We estimated that 47% (SE = 3.9%) of the P-values in the Journal of Wildlife Management lacked estimates of means or effect sizes or even the sign of the difference in means or other parameters. We find that null hypothesis testing is uninformative when no estimates of means or effect size and their precision are given. Contrary to common dogma, tests of statistical null hypotheses have relatively little utility in science and are not a fundamental aspect of the scientific method. We recommend their use be reduced in favor of more informative approaches. Towards this objective, we describe a relatively new paradigm of data analysis based on Kullback-Leibler information. This paradigm is an extension of likelihood theory and, when used correctly, avoids many of the fundamental limitations and common misuses of null hypothesis testing. Information-theoretic methods focus on providing a strength of evidence for an a priori set of alternative hypotheses, rather than a statistical test of a null hypothesis. This paradigm allows the following types of evidence for the alternative hypotheses: the rank of each hypothesis, expressed as a model; an estimate of the formal likelihood of each model, given the data; a measure of precision that incorporates model selection uncertainty; and simple methods to allow the use of the set of alternative models in making, formal inference. We provide an example of the information-theoretic approach using data on the effect of lead on survival in spectacled eider ducks (Somateria fischeri). Regardless of the analysis paradigm used, we strongly recommend inferences based on a priori considerations be clearly separated from those resulting from some form of data dredging.

Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries

The application of null hypothesis testing to psychological research has been much criticized (e.g., Bakan, 1966; Gigerenzer & Murray, 1987). I examine the specific relevance of these and other criticisms for research on sex differences. Four specific problems are identified: (1) drawing inferences about general properties that are attributed to all members of a population; (2) the distinction between the size of p and the size and theoretical importance of a difference; (3) the frequently unjustified assumption of normality; and (4) the semantic problems inherent in the language of interpretation. A few solutions are explored, and it is concluded that descriptions of the results of sex comparisons, as well as others, must reflect the data more accurately than they now do. A number of years ago, in an invited address that I gave at the annual meeting of the Canadian Psychological Association, I criticized the way in which standard null hypothesis statistics are used in research on sex differences. I argued that in just about every known comparison of the behaviour of females and males there is generally a substantial amount of overlap between the distributions of the dependent variable for the members of each sex (if the behaviour in question is physically possible for both). This is true even if a statistical test has led to the inference that the difference is significant (p .01, p Notwithstanding such extreme cases of overlap, the mere fact that any overlap occurs at all, I reasoned, renders logically false the descriptive statements that are made when differences are said to be significant. For example, to cite a currently contentious issue, suppose it is found that the average score obtained by boys on a math test is significantly higher than that obtained by girls. Since these scores invariably overlap, (see, e.g., Benbow, 1988, p. 219; Hyde et. al., 1990), then it is false to translate the statistical result into an affirmation which states that boys are better at math than are girls. I should perhaps add that this logic also applies when girls obtain higher average scores than boys, a result which has been observed in a larger variety of situations than seems to be commonly known (Kimball, 1989).During the question period, I was asked how it is possible to make a special case that null hypothesis testing is inappropriate for sex differences, when these tests were in general use for most other kinds of psychological data. My questioner clearly did not mean to imply that significance testing should be discarded altogether (or so I understood him) but rather, since these types of statistical procedures are the standard tools of contemporary experimental psychology, and therefore must be valid, then it could hardly be legitimate to make a special case for their irrelevance to sex differences. To my embarrassment, I was unable at that time to give a satisfactory reply, neither to the questioner nor to myself. On the one hand, I was convinced that my reasoning was correct as far as sex differences are concerned, yet, on the other hand, I was still so attached to significance tests as a general tool for psychological research that I was not yet ready to think that their application could be more generally problematical.Over the intervening years I have discovered a rather substantial literature that is critical of the ways in which null hypothesis testing has been integrated into the entire psychological research process. …

Many empirical studies estimating effective population size apply the temporal method that provides an estimate of the variance effective size through the amount of temporal allele frequency change under the assumption that the study population is completely isolated. This assumption is frequently violated, and the magnitude of the resulting bias is generally unknown. We studied how gene flow affects estimates of effective size obtained by the temporal method when sampling from a population system and provide analytical expressions for the expected estimate under an island model of migration. We show that the temporal method tends to systematically underestimate both local and global effective size when populations are connected by gene flow, and the bias is sometimes dramatic. The problem is particularly likely to occur when sampling from a subdivided population where high levels of gene flow obscure identification of subpopulation boundaries. In such situations, sampling in a manner that prevents biased estimates can be difficult. This phenomenon might partially explain the frequently reported unexpectedly low effective population sizes of marine populations that have raised concern regarding the genetic vulnerability of even exceptionally large populations.

We examined the degree of autocorrelation among single‐case design data with six measures used to estimate treatment effect size. The most commonly used measures of effect size for single‐case data over the last 5 years published in peer‐reviewed journals were selected for comparison. The overall mean degree of autocorrelation was 0.46 (SD = 0.33) across the 304 data paths, which represents a moderate degree of autocorrelation. Overall, it appears that non‐parametric measures of effect size (i.e., percent of non‐overlapping data [PND], non‐overlap of all pairs [NAP], and improvement rate difference [IRD] values) were substantially and significantly more influenced by the degree of autocorrelation. Tau‐U effect size estimate was the non‐parametric exception as it was not significantly influenced by the degree of autocorrelation. Parametric measures of effect sizes such as standardized mean difference (SMD) and log response ratio (LRR) values did not appear to be significantly influenced by the degree of autocorrelation. For SMD, LRR, and Tau‐U values, the correlation between the effect size value and the degree of autocorrelation was minimal. For NAP, IRD, and PND values, the correlation between the effect size value and the degree of autocorrelation was moderate, indicating that these estimates of effect size should be avoided as the degree of autocorrelation between data points increases.

Expanding the toxicologist's statistical toolbox: Using effect size estimation and dose-response modelling for holistic assessments instead of generic testing

This paper offers a practical guide to use null hypotheses significance testing and its alternatives. The focus is on improving the quality of statistical inference in quantitative communication research. More consistent reporting of descriptive statistics, estimates of effect size, confidence intervals around effect sizes, and increasing the statistical power of tests would lead to needed improvements over current practices. Alternatives including confidence intervals, effect tests, equivalence tests, and meta-analysis are discussed.

The ease with which data can be collected and analyzed via personal computer makes it potentially attractive to "peek" at the data before a target sample size is achieved. This tactic might seem appealing because data collection could be stopped early, which would save valuable resources, if a peek revealed a significant effect. Unfortunately, such data snooping comes with a cost. When the null hypothesis is true, the Type I error rate is inflated, sometimes quite substantially. If the null hypothesis is false, premature significance testing leads to inflated estimates of power and effect size. This program provides simulation results for a wide variety of premature and repeated null hypothesis testing scenarios. It gives researchers the ability to know in advance the consequences of data peeking so that appropriate corrective action can be taken.

The power of statistical tests used in behavioral research has been of concern since Cohen's (1962) indictment of abnormal-social psychological research. His findings of minimal power values of statistical tests in published research has been replicated by Brewer (1972) in educational research and by Chase and Chase (1976) in applied psychology. Perhaps one reason behavioral scientists have been reticent to employ power analysis when designing and reporting research is the lack of a ready technology for accomplishing this goal. This void led Cohen (1969, 1977) to write two books on statistical power analysis that contain extensive tables for power determination. Errors in his 1969 book acknowledged by Cohen (1977) and the unwieldy nature of his tables and their usage led us to write POWERF, based on the earlier work of Borich and Godbout (1974). To briefly review terminology: Alpha (a) is the probability of committing a Type I error, and 1 a is the degree of confidence one has that a statistically significant difference is not due to random error. Beta (~) is the probability of committing a Type II error. The power of a test is 1 -~, or the probability of rejecting the null hypothesis when the sample means come from different populations. Program POWERF employs an algorithm reported by Cohen (1977) to compute the power of F tests of fixed main effects or F tests of interactions of fixed-effects factors in ANOVA designs. POWERF is written in CDC FORTRAN and is a fast running program. On the dual Cyber 170/750 system at the University of Texas, 36 power determinations were computed in .199 CPU sec. Input. Four or five values are input, depending on program usage. When estimating the power of a planned experiment, one must input the tabled F value required for significance at any desired a level, the degrees of freedom (df) in the F ratio numerator, the df in the F ratio denominator, and the estimated effect size (ES). The ES is a measure of the degree to which the independent variable or treatment of interest affects the dependent variable. In planning an experiment, ES may be estimated from prior research results or the convention may be employed that .10 is a small, .25 is a medium, and .40 is a large ES (Cohen, 1977). When evaluating a completed study, ES may be computed (ES: n n , where n : eta, the correlation ratio, squared). If ES is left blank on the input card, a second card will be read. This card must have either eta squared punched in the first four columns or the sum of squares for the main or interaction effect of interest and the sum of squares total, respectively, punched under a (4X, 2F13.3) format. For a more detailed discussion of ES, see Cohen (I977) and Friedman (I 968). Output. Five values are printed for each set of input values. First, the tabled F value that was input is printed. Next the ES, cell n, df numerator, df denominator, and power are printed. The n value was used by Cohen (I977) in constructing his power tables and will allow one to compare the results of POWERF with Cohen's tabled values. This comparison in fact was made for a total of 36 problems formed by combinations of a: .01 or .05, ES:.10 or .40, numerator df: 1, 3, or 10, and n (the average cell size): 2, 10, or 50. When the power value was rounded to two places, POWERF produced values that were equal to Cohen's (1977) tabled values in 25 of 36 instances. In all nine cases in which POWERF produced disparate values, they varied only .01 from the tabled values. Availability. A listing of POWERF may be obtained free of charge to those providing a self-addressed, stamped envelope from Randall Parker, PhD, Department of Special Education, EDB 306, University of Texas, Austin, Texas 78712.

Wilcoxon-Mann-Whitney odds ratio: A statistical measure for ordinal outcomes such as EDSS

Confirming the Validity of the General Pediatrics Certification Examinations: A Practice Analysis

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

Kühberger, Scherndl, and Fritz commented on an attempt by Bradley and Brand to adjust sets of exaggerated effect sizes reported in literatures associated with underpowered Null Hypothesis Statistical Tests (NHST). Their comment highlighted two important issues: (a) the senior author, Bradley, made an error in presenting the correction formula, and (b) there is an inherent incompatibility between inferential statistics and accurate measurement. The proper formula is presented here with evidence that the formula is relatively accurate in appropriately estimating effect sizes that have been exaggerated through NHST. The term relatively accurate is used since power cannot be 100%, and thus any attempt to estimate a true effect size will be out by some relationship between the alpha level, power, and of course statistical variability of the estimates.

The dialog surrounding effect sizes and statistical significance tests often places the two ideas into separate camps amid controversy. In light of recommendations by the Task Force on Statistical Inference and the fifth edition of the American Psychological Association Publication Manual calling for the reporting of effect sizes, a review of treatments of effect sizes in textbooks may be quite timely. This study reviews textbooks published since 1995 and as regards treatments of effect sizes and statistical significance tests. Of The textbooks examined, every textbook ( n = 89) included the topic of statistical significance testing (2,248 pages), whereas only a little more than two thirds of the textbooks ( n = 60) included information on effect sizes (789 pages).