Null Hypothesis Testing: Problems, Prevalence, and an Alternative

REX B. KLINE Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research Washington, DC: American Psychological Association, 2004, 336 pages (ISBN 1-59147-118-4, US$49.95 Hardcover) In 1999, a blue-ribbon task force assembled by the American Psychological Association published their findings with regards to the long-standing controversy pertaining to null hypothesis testing (NHST). The task force dictated effect sizes and confidence intervals be reported, and p values and dichotomous accept-reject decisions be given less weight. Editorial policies in a number of journals came to reflect the views of the task force as did a subsequent revision to the American Psychological Association Publication Manual. Rex B. Kline wrote Beyond Significance Testing. Reforming Data Analysis Methods in Behavioral Research as a follow-up to both the task force recommendations and the revision to the publication manual. Kline's 1998 book Principles and Practice of Structural Equation Modeling (Guilford Press) was well received and a second edition is being published this fall. In Beyond Significance Testing, Kline reviews the controversy regarding testing, offers methods for effect size and confidence interval estimation, and suggests some methodologies. There is an accompanying website that includes resources for instructors and students. Part I of the book is a review of fundamental concepts and the debate regarding testing. Part II provides statistics for effect size and confidence interval estimation for parametric and nonparametric two-group, oneway, and factorial designs. Part III examines metaanalysis, resampling, and Bayesian estimation procedures. In the first chapter, Kline provides a scholarly summary of the null hypothesis testing debate concluding with the APA task force findings and what Kline regards as ambiguous recommendations in the publication manual. Kline predicts the future will see a smaller role for traditional statistical testing (p values) in psychology. This change will take time and may not occur until the next generation of researchers are trained, but Kline anticipates the social sciences will then become more like the natural sciences in that we will report the directions and magnitudes of our effects, determine whether they replicate, and evaluate them for their theoretical, clinical, or practical significance (p. 15). Chapter 2 is a review of fundamental concepts of research design, including sampling and estimation, the logic of statistical testing, and t, F, and chi-square tests. The problems with statistical tests are revisited in Chapter 3. What follows is a long list of errors in interpretation of p values and conclusions made after null hypothesis testing. The emphasis on null hypothesis testing in psychology is also argued to inhibit advancement of the discipline. To be fair, Kline recognizes there is yet no magical alternative to statistical tests and that such tests are appropriate in some circumstances when applied correctly. Nonetheless, Kline envisions a future where effect sizes and confidence intervals are reported, substantive rather than statistical predominates, and NHST-Centric thinking has diminished. Part II covers effect size and confidence interval calculations. Chapter 4 is a presentation of parametric effect size indexes. Independent and dependent sample statistics are covered separately. The textbook's website has a supplementary chapter on twogroup multivariate designs. Group difference indexes such as d are distinguished from measures of association such as r. Case level analyses of group differences are also reviewed. Sections not relevant to a reader's needs can be skipped without loss of continuity. Interpretive guidelines for effect size magnitude and how one might be fooled by effect size estimation are sections that should not be passed over. …

Chapter Seven - The debate over null hypothesis testing and wise use as the solution

The application of null hypothesis testing to psychological research has been much criticized (e.g., Bakan, 1966; Gigerenzer & Murray, 1987). I examine the specific relevance of these and other criticisms for research on sex differences. Four specific problems are identified: (1) drawing inferences about general properties that are attributed to all members of a population; (2) the distinction between the size of p and the size and theoretical importance of a difference; (3) the frequently unjustified assumption of normality; and (4) the semantic problems inherent in the language of interpretation. A few solutions are explored, and it is concluded that descriptions of the results of sex comparisons, as well as others, must reflect the data more accurately than they now do. A number of years ago, in an invited address that I gave at the annual meeting of the Canadian Psychological Association, I criticized the way in which standard null hypothesis statistics are used in research on sex differences. I argued that in just about every known comparison of the behaviour of females and males there is generally a substantial amount of overlap between the distributions of the dependent variable for the members of each sex (if the behaviour in question is physically possible for both). This is true even if a statistical test has led to the inference that the difference is significant (p .01, p Notwithstanding such extreme cases of overlap, the mere fact that any overlap occurs at all, I reasoned, renders logically false the descriptive statements that are made when differences are said to be significant. For example, to cite a currently contentious issue, suppose it is found that the average score obtained by boys on a math test is significantly higher than that obtained by girls. Since these scores invariably overlap, (see, e.g., Benbow, 1988, p. 219; Hyde et. al., 1990), then it is false to translate the statistical result into an affirmation which states that boys are better at math than are girls. I should perhaps add that this logic also applies when girls obtain higher average scores than boys, a result which has been observed in a larger variety of situations than seems to be commonly known (Kimball, 1989).During the question period, I was asked how it is possible to make a special case that null hypothesis testing is inappropriate for sex differences, when these tests were in general use for most other kinds of psychological data. My questioner clearly did not mean to imply that significance testing should be discarded altogether (or so I understood him) but rather, since these types of statistical procedures are the standard tools of contemporary experimental psychology, and therefore must be valid, then it could hardly be legitimate to make a special case for their irrelevance to sex differences. To my embarrassment, I was unable at that time to give a satisfactory reply, neither to the questioner nor to myself. On the one hand, I was convinced that my reasoning was correct as far as sex differences are concerned, yet, on the other hand, I was still so attached to significance tests as a general tool for psychological research that I was not yet ready to think that their application could be more generally problematical.Over the intervening years I have discovered a rather substantial literature that is critical of the ways in which null hypothesis testing has been integrated into the entire psychological research process. …

Hypothesis tests

We give suggestions for the presentation of research results from frequentist, information-theoretic, and Bayesian analysis paradigms, followed by several general suggestions. The information-theoretic and Bayesian methods offer alternative approaches to data analysis and inference compared to traditionally used methods. Guidance is lacking on the presentation of results under these alternative procedures and on nontesting aspects of classical frequentist methods of statistical analysis. Null hypothesis testing has come under intense criticism. We recommend less reporting of the results of statistical tests of null hypotheses in cases where the null is surely false anyway, or where the null hypothesis is of little interest to science or management.

The ease with which data can be collected and analyzed via personal computer makes it potentially attractive to "peek" at the data before a target sample size is achieved. This tactic might seem appealing because data collection could be stopped early, which would save valuable resources, if a peek revealed a significant effect. Unfortunately, such data snooping comes with a cost. When the null hypothesis is true, the Type I error rate is inflated, sometimes quite substantially. If the null hypothesis is false, premature significance testing leads to inflated estimates of power and effect size. This program provides simulation results for a wide variety of premature and repeated null hypothesis testing scenarios. It gives researchers the ability to know in advance the consequences of data peeking so that appropriate corrective action can be taken.

FOUNDATIONS Philosophical and Historical Foundations Introduction Nature of Science Scientific Principles Scientific Method Scientific Hypotheses Logic Variability and Uncertainty in Investigations Science and Statistics Statistics and Biology Introduction to Probability Introduction: Models for Random Variables Classical Probability Conditional Probability Odds Combinatorial Analysis Bayes Rule Probability Density Functions Introduction Introductory Examples of pdfs Other Important Distributions Which pdf to Use? Reference Tables Parameters and Statistics Introduction Parameters Statistics OLS and ML Estimators Linear Transformations Bayesian Applications Interval Estimation: Sampling Distributions, Resampling Distributions, and Simulation Distributions Introduction Sampling Distributions Confidence Intervals Resampling Distributions Bayesian Applications: Simulation Distributions Hypothesis Testing Introduction Parametric Frequentist Null Hypothesis Testing Type I and Type II Errors Power Criticisms of Frequentist Null Hypothesis Testing Alternatives to Parametric Null Hypothesis Testing Alternatives to Null Hypothesis Testing Sampling Design and Experimental Design Introduction Some Terminology The Question Is: What Is the Question? Two Important Tenets: Randomization and Replication Sampling Design Experimental Design APPLICATIONS Correlation Introduction Pearson's Correlation Robust Correlation Comparisons of Correlation Procedures Regression Introduction Linear Regression Model General Linear Models Simple Linear Regression Multiple Regression Fitted and Predicted Values Confidence and Prediction Intervals Coefficient of Determination and Important Variants Power, Sample Size, and Effect Size Assumptions and Diagnostics for Linear Regression Transformation in the Context of Linear Models Fixing the Y-Intercept Weighted Least Squares Polynomial Regression Comparing Model Slopes Likelihood and General Linear Models Model Selection Robust Regression Model II Regression (X Not Fixed) Generalized Linear Models Nonlinear Models Smoother Approaches to Association and Regression Bayesian Approaches to Regression ANOVA Introduction One-Way ANOVA Inferences for Factor Levels ANOVA as a General Linear Model Random Effects Power, Sample Size, and Effect Size ANOVA Diagnostics and Assumptions Two-Way Factorial Design Randomized Block Design Nested Design Split-Plot Design Repeated Measures Design ANCOVA Unbalanced Designs Robust ANOVA Bayesian Approaches to ANOVA Tabular Analyses Introduction Probability Distributions for Tabular Analyses One-Way Formats Confidence Intervals for p Contingency Tables Two-Way Tables Ordinal Variables Power, Sample Size, and Effect Size Three-Way Tables Generalized Linear Models Appendix References Index A Summary and Exercises appear at the end of each chapter.

The Hubble parameter is a critical measurement in cosmology, which contains the most direct information of the cosmic expansion history. Since discrepancy is found between low redshift and high redshift estimations of Hubble constant, we are interested in whether that tension indicates dynamical dark energy. In this paper, we emphasize that the observed Hubble parameters at various redshifts, along with observed Hubble constant, can help us in probing the evolutional behavior of the mysterious dark energy. Null hypothesis tests are carried out with two diagnostic approaches. We find out that, according to the present measurements of Hubble parameters, rejection of constant dark energy is captured at 1σ level from null tests with and without the observed value of Hubble constant.

Null hypothesis testing has long since been the ‘go‐to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open‐science framework, the present study (a) re‐evaluates the empirical findings of 418 t‐tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p‐values fell between .01 and .05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field‐wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.

Summary Stephens et al. (2005) argue for ‘pluralism’ in statistical analysis, combining null hypothesis testing and information‐theoretic (I‐T) methods. We show that I‐T methods are more informative even in single variable problems and we provide an ecological example. I‐T methods allow inferences to be made from multiple models simultaneously. We believe multimodel inference is the future of data analysis, which cannot be achieved with null hypothesis‐testing approaches. We argue for a stronger emphasis on critical thinking in science in general and less reliance on exploratory data analysis and data dredging. Deriving alternative hypotheses is central to science; deriving a single interesting science hypothesis and then comparing it to a default null hypothesis (e.g. ‘no difference’) is not an efficient strategy for gaining knowledge. We think this single‐hypothesis strategy has been relied upon too often in the past. We clarify misconceptions presented by Stephens et al. (2005). We think inference should be made about models, directly linked to scientific hypotheses, and their parameters conditioned on data, Prob(Hj | data). I‐T methods provide a basis for this inference. Null hypothesis testing merely provides a probability statement about the data conditioned on a null model, Prob(data | H0). Synthesis and applications. I‐T methods provide a more informative approach to inference. I‐T methods provide a direct measure of evidence for or against hypotheses and a means to consider simultaneously multiple hypotheses as a basis for rigorous inference. Progress in our science can be accelerated if modern methods can be used intelligently; this includes various I‐T and Bayesian methods.

Experimental power comes from powerful theories — the real problem in null hypothesis testing

[This corrects the article DOI: 10.1371/journal.pone.0222908.].

치의학 연구에서 사용되는 귀무가설 유의성 검정에서 p값을 기준으로 연구의 결과를 평가하는 것은 많은 문제점을 내포하고 있다. 귀무가설이 기각되지 않은 경우에 귀무가설이 옳다는 결론을 내리는 것은 논리적 오류이다. p값에 대한 중대한 오해가 많이 있으며 연구자는 논문을 작성할 때 p값의 해석에 신중해야 한다. 귀무가설검정을 보완하거나 대체할 수 있는 대안으로서, 효과 크기, 신뢰구간, 베이지안 통계 등이 있다. There are many problems in evaluating study results by p value in null hypothesis testing for dental research. It is a logical fallacy to conclude that the null hypothesis is true when the it is not rejected. There are much serious misunderstanding about p value, and researchers should be cautious about interpreting p value in writing papers. As alternatives to complement or replace the null hypothesis significance testing, effect size, confidence interval, and Bayesian statistics are introduced.

Individual-based analyses relating landscape structure to genetic distances across complex landscapes enable rigorous evaluation of multiple alternative hypotheses linking landscape structure to gene flow. We utilize two extensions to increase the rigor of the individual-based causal modeling approach to inferring relationships between landscape patterns and gene flow processes. First, we add a univariate scaling analysis to ensure that each landscape variable is represented in the functional form that represents the optimal scale of its association with gene flow. Second, we use a two-step form of the causal modeling approach to integrate model selection with null hypothesis testing in individual-based landscape genetic analysis. This series of causal modeling indicated that gene flow in American marten in northern Idaho was primarily related to elevation, and that alternative hypotheses involving isolation by distance, geographical barriers, effects of canopy closure, roads, tree size class and an empirical habitat model were not supported. Gene flow in the Northern Idaho American marten population is therefore driven by a gradient of landscape resistance that is a function of elevation, with minimum resistance to gene flow at 1500 m.

Agronomic research results include Type 1 (α) and Type 2 (β) errors. Results are often reported using α ≤ 0.05 while β is ignored. Our objective was to discuss whether a false positive was more serious than a false negative in agronomic research. For comparison, current statistical methods used in Agronomy Journal were tabulated. Most papers used null hypothesis tests with α ≤ 0.05, reporting results based on the LSD among all treatment pairs. Current practices do not account for the relative costs of false positive vs. false negative errors. A case study from the Washington State Wheat Extension trials was analyzed using mixed models with specific contrasts. While the overall effect for cultivar was significant, the β error rate for the contrast was 40% and additional replications were needed to increase the power of this contrast. A second case study analyzed trials evaluating wheat (Triticum aestivum L.) resistance to Fusarium crown rot. Optimal α and β error rates were estimated for two to eight replications with the Type1/Type2 error cost ratio set at 1:1 and 1:5. An average error rate (α and β) ≤ 0.05 could be achieved with four replications when a reduction in the β error was critical and α errors could be corrected in future experiments. Effective experimental design requires estimation of the acceptable magnitude and cost ratio of false positive and false negative errors and critical effect sizes. To be truly informative, reports of results should include this information plus observed effect sizes and variances.