Chapter Seven - The debate over null hypothesis testing and wise use as the solution

REX B. KLINE Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research Washington, DC: American Psychological Association, 2004, 336 pages (ISBN 1-59147-118-4, US$49.95 Hardcover) In 1999, a blue-ribbon task force assembled by the American Psychological Association published their findings with regards to the long-standing controversy pertaining to null hypothesis testing (NHST). The task force dictated effect sizes and confidence intervals be reported, and p values and dichotomous accept-reject decisions be given less weight. Editorial policies in a number of journals came to reflect the views of the task force as did a subsequent revision to the American Psychological Association Publication Manual. Rex B. Kline wrote Beyond Significance Testing. Reforming Data Analysis Methods in Behavioral Research as a follow-up to both the task force recommendations and the revision to the publication manual. Kline's 1998 book Principles and Practice of Structural Equation Modeling (Guilford Press) was well received and a second edition is being published this fall. In Beyond Significance Testing, Kline reviews the controversy regarding testing, offers methods for effect size and confidence interval estimation, and suggests some methodologies. There is an accompanying website that includes resources for instructors and students. Part I of the book is a review of fundamental concepts and the debate regarding testing. Part II provides statistics for effect size and confidence interval estimation for parametric and nonparametric two-group, oneway, and factorial designs. Part III examines metaanalysis, resampling, and Bayesian estimation procedures. In the first chapter, Kline provides a scholarly summary of the null hypothesis testing debate concluding with the APA task force findings and what Kline regards as ambiguous recommendations in the publication manual. Kline predicts the future will see a smaller role for traditional statistical testing (p values) in psychology. This change will take time and may not occur until the next generation of researchers are trained, but Kline anticipates the social sciences will then become more like the natural sciences in that we will report the directions and magnitudes of our effects, determine whether they replicate, and evaluate them for their theoretical, clinical, or practical significance (p. 15). Chapter 2 is a review of fundamental concepts of research design, including sampling and estimation, the logic of statistical testing, and t, F, and chi-square tests. The problems with statistical tests are revisited in Chapter 3. What follows is a long list of errors in interpretation of p values and conclusions made after null hypothesis testing. The emphasis on null hypothesis testing in psychology is also argued to inhibit advancement of the discipline. To be fair, Kline recognizes there is yet no magical alternative to statistical tests and that such tests are appropriate in some circumstances when applied correctly. Nonetheless, Kline envisions a future where effect sizes and confidence intervals are reported, substantive rather than statistical predominates, and NHST-Centric thinking has diminished. Part II covers effect size and confidence interval calculations. Chapter 4 is a presentation of parametric effect size indexes. Independent and dependent sample statistics are covered separately. The textbook's website has a supplementary chapter on twogroup multivariate designs. Group difference indexes such as d are distinguished from measures of association such as r. Case level analyses of group differences are also reviewed. Sections not relevant to a reader's needs can be skipped without loss of continuity. Interpretive guidelines for effect size magnitude and how one might be fooled by effect size estimation are sections that should not be passed over. …

This paper presents a review and critique of statistical null hypothesis testing in ecological studies in general, and wildlife studies in particular, and describes an alternative. Our review of Ecology and the Journal of Wildlife Management found the use of null hypothesis testing to be pervasive. The estimated number of P-values appearing within articles of Ecology exceeded 8,000 in 1991 and has exceeded 3,000 in each year since 1984, whereas the estimated number of P-values in the Journal of Wildlife Management exceeded 8,000 in 1997 and has exceeded 3,000 in each year since 1994. We estimated that 47% (SE = 3.9%) of the P-values in the Journal of Wildlife Management lacked estimates of means or effect sizes or even the sign of the difference in means or other parameters. We find that null hypothesis testing is uninformative when no estimates of means or effect size and their precision are given. Contrary to common dogma, tests of statistical null hypotheses have relatively little utility in science and are not a fundamental aspect of the scientific method. We recommend their use be reduced in favor of more informative approaches. Towards this objective, we describe a relatively new paradigm of data analysis based on Kullback-Leibler information. This paradigm is an extension of likelihood theory and, when used correctly, avoids many of the fundamental limitations and common misuses of null hypothesis testing. Information-theoretic methods focus on providing a strength of evidence for an a priori set of alternative hypotheses, rather than a statistical test of a null hypothesis. This paradigm allows the following types of evidence for the alternative hypotheses: the rank of each hypothesis, expressed as a model; an estimate of the formal likelihood of each model, given the data; a measure of precision that incorporates model selection uncertainty; and simple methods to allow the use of the set of alternative models in making, formal inference. We provide an example of the information-theoretic approach using data on the effect of lead on survival in spectacled eider ducks (Somateria fischeri). Regardless of the analysis paradigm used, we strongly recommend inferences based on a priori considerations be clearly separated from those resulting from some form of data dredging.

The application of null hypothesis testing to psychological research has been much criticized (e.g., Bakan, 1966; Gigerenzer & Murray, 1987). I examine the specific relevance of these and other criticisms for research on sex differences. Four specific problems are identified: (1) drawing inferences about general properties that are attributed to all members of a population; (2) the distinction between the size of p and the size and theoretical importance of a difference; (3) the frequently unjustified assumption of normality; and (4) the semantic problems inherent in the language of interpretation. A few solutions are explored, and it is concluded that descriptions of the results of sex comparisons, as well as others, must reflect the data more accurately than they now do. A number of years ago, in an invited address that I gave at the annual meeting of the Canadian Psychological Association, I criticized the way in which standard null hypothesis statistics are used in research on sex differences. I argued that in just about every known comparison of the behaviour of females and males there is generally a substantial amount of overlap between the distributions of the dependent variable for the members of each sex (if the behaviour in question is physically possible for both). This is true even if a statistical test has led to the inference that the difference is significant (p .01, p Notwithstanding such extreme cases of overlap, the mere fact that any overlap occurs at all, I reasoned, renders logically false the descriptive statements that are made when differences are said to be significant. For example, to cite a currently contentious issue, suppose it is found that the average score obtained by boys on a math test is significantly higher than that obtained by girls. Since these scores invariably overlap, (see, e.g., Benbow, 1988, p. 219; Hyde et. al., 1990), then it is false to translate the statistical result into an affirmation which states that boys are better at math than are girls. I should perhaps add that this logic also applies when girls obtain higher average scores than boys, a result which has been observed in a larger variety of situations than seems to be commonly known (Kimball, 1989).During the question period, I was asked how it is possible to make a special case that null hypothesis testing is inappropriate for sex differences, when these tests were in general use for most other kinds of psychological data. My questioner clearly did not mean to imply that significance testing should be discarded altogether (or so I understood him) but rather, since these types of statistical procedures are the standard tools of contemporary experimental psychology, and therefore must be valid, then it could hardly be legitimate to make a special case for their irrelevance to sex differences. To my embarrassment, I was unable at that time to give a satisfactory reply, neither to the questioner nor to myself. On the one hand, I was convinced that my reasoning was correct as far as sex differences are concerned, yet, on the other hand, I was still so attached to significance tests as a general tool for psychological research that I was not yet ready to think that their application could be more generally problematical.Over the intervening years I have discovered a rather substantial literature that is critical of the ways in which null hypothesis testing has been integrated into the entire psychological research process. …

We give suggestions for the presentation of research results from frequentist, information-theoretic, and Bayesian analysis paradigms, followed by several general suggestions. The information-theoretic and Bayesian methods offer alternative approaches to data analysis and inference compared to traditionally used methods. Guidance is lacking on the presentation of results under these alternative procedures and on nontesting aspects of classical frequentist methods of statistical analysis. Null hypothesis testing has come under intense criticism. We recommend less reporting of the results of statistical tests of null hypotheses in cases where the null is surely false anyway, or where the null hypothesis is of little interest to science or management.

FOUNDATIONS Philosophical and Historical Foundations Introduction Nature of Science Scientific Principles Scientific Method Scientific Hypotheses Logic Variability and Uncertainty in Investigations Science and Statistics Statistics and Biology Introduction to Probability Introduction: Models for Random Variables Classical Probability Conditional Probability Odds Combinatorial Analysis Bayes Rule Probability Density Functions Introduction Introductory Examples of pdfs Other Important Distributions Which pdf to Use? Reference Tables Parameters and Statistics Introduction Parameters Statistics OLS and ML Estimators Linear Transformations Bayesian Applications Interval Estimation: Sampling Distributions, Resampling Distributions, and Simulation Distributions Introduction Sampling Distributions Confidence Intervals Resampling Distributions Bayesian Applications: Simulation Distributions Hypothesis Testing Introduction Parametric Frequentist Null Hypothesis Testing Type I and Type II Errors Power Criticisms of Frequentist Null Hypothesis Testing Alternatives to Parametric Null Hypothesis Testing Alternatives to Null Hypothesis Testing Sampling Design and Experimental Design Introduction Some Terminology The Question Is: What Is the Question? Two Important Tenets: Randomization and Replication Sampling Design Experimental Design APPLICATIONS Correlation Introduction Pearson's Correlation Robust Correlation Comparisons of Correlation Procedures Regression Introduction Linear Regression Model General Linear Models Simple Linear Regression Multiple Regression Fitted and Predicted Values Confidence and Prediction Intervals Coefficient of Determination and Important Variants Power, Sample Size, and Effect Size Assumptions and Diagnostics for Linear Regression Transformation in the Context of Linear Models Fixing the Y-Intercept Weighted Least Squares Polynomial Regression Comparing Model Slopes Likelihood and General Linear Models Model Selection Robust Regression Model II Regression (X Not Fixed) Generalized Linear Models Nonlinear Models Smoother Approaches to Association and Regression Bayesian Approaches to Regression ANOVA Introduction One-Way ANOVA Inferences for Factor Levels ANOVA as a General Linear Model Random Effects Power, Sample Size, and Effect Size ANOVA Diagnostics and Assumptions Two-Way Factorial Design Randomized Block Design Nested Design Split-Plot Design Repeated Measures Design ANCOVA Unbalanced Designs Robust ANOVA Bayesian Approaches to ANOVA Tabular Analyses Introduction Probability Distributions for Tabular Analyses One-Way Formats Confidence Intervals for p Contingency Tables Two-Way Tables Ordinal Variables Power, Sample Size, and Effect Size Three-Way Tables Generalized Linear Models Appendix References Index A Summary and Exercises appear at the end of each chapter.

Null hypothesis testing has long since been the ‘go‐to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open‐science framework, the present study (a) re‐evaluates the empirical findings of 418 t‐tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p‐values fell between .01 and .05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field‐wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.

The Hubble parameter is a critical measurement in cosmology, which contains the most direct information of the cosmic expansion history. Since discrepancy is found between low redshift and high redshift estimations of Hubble constant, we are interested in whether that tension indicates dynamical dark energy. In this paper, we emphasize that the observed Hubble parameters at various redshifts, along with observed Hubble constant, can help us in probing the evolutional behavior of the mysterious dark energy. Null hypothesis tests are carried out with two diagnostic approaches. We find out that, according to the present measurements of Hubble parameters, rejection of constant dark energy is captured at 1σ level from null tests with and without the observed value of Hubble constant.

The ease with which data can be collected and analyzed via personal computer makes it potentially attractive to "peek" at the data before a target sample size is achieved. This tactic might seem appealing because data collection could be stopped early, which would save valuable resources, if a peek revealed a significant effect. Unfortunately, such data snooping comes with a cost. When the null hypothesis is true, the Type I error rate is inflated, sometimes quite substantially. If the null hypothesis is false, premature significance testing leads to inflated estimates of power and effect size. This program provides simulation results for a wide variety of premature and repeated null hypothesis testing scenarios. It gives researchers the ability to know in advance the consequences of data peeking so that appropriate corrective action can be taken.

This study aims to determine the effect of the inquiry learning model and the creative problem-solving learning model on student learning outcomes. This research was conducted at MTs Madinatussalam Tembung for the academic year 2020/2021. The research method is quasi-experimental. The sample used in the study was 60 students, the sampling in this study was simple random sampling. The instrument in the study used a pretest and posttest. Based on the results of the research, the results of hypothesis testing on the results of the Inquiry class hypothesis test. From the table above, the value of t(count )= 8,594 and the value of t_(table )= 2.0243 so that t_(count)>t_(table) which is therefore Ho is rejected. Ha accepted. Based on the results of hypothesis testing, there is an influence of the inquiry learning model. The results of the hypothesis test for the Creative Problem-Solving class. From the table above, the value of t(count )= 8, 683 and the value of t_(table )= 2.0243 so that t_(count )>t_(table ) which means that Ho is rejected, Ha is accepted. Based on the results of hypothesis testing, there is an influence of the Inquiry learning model and Creative Problem-Solving learning model on student learning outcomes at MTs Madinatussalam Medan Tembung.

Summary Stephens et al. (2005) argue for ‘pluralism’ in statistical analysis, combining null hypothesis testing and information‐theoretic (I‐T) methods. We show that I‐T methods are more informative even in single variable problems and we provide an ecological example. I‐T methods allow inferences to be made from multiple models simultaneously. We believe multimodel inference is the future of data analysis, which cannot be achieved with null hypothesis‐testing approaches. We argue for a stronger emphasis on critical thinking in science in general and less reliance on exploratory data analysis and data dredging. Deriving alternative hypotheses is central to science; deriving a single interesting science hypothesis and then comparing it to a default null hypothesis (e.g. ‘no difference’) is not an efficient strategy for gaining knowledge. We think this single‐hypothesis strategy has been relied upon too often in the past. We clarify misconceptions presented by Stephens et al. (2005). We think inference should be made about models, directly linked to scientific hypotheses, and their parameters conditioned on data, Prob(Hj | data). I‐T methods provide a basis for this inference. Null hypothesis testing merely provides a probability statement about the data conditioned on a null model, Prob(data | H0). Synthesis and applications. I‐T methods provide a more informative approach to inference. I‐T methods provide a direct measure of evidence for or against hypotheses and a means to consider simultaneously multiple hypotheses as a basis for rigorous inference. Progress in our science can be accelerated if modern methods can be used intelligently; this includes various I‐T and Bayesian methods.

Experimental power comes from powerful theories — the real problem in null hypothesis testing

This research is an attempt to study the relationship between audit firm rotation and audit reports in companies listed on the Tehran Stock Exchange from 2003 to 2011. Due to the nominal nature of the variables, this study uses a non-parametric chi square distribution test to analyse the collected data. The results indicate that there is a significant relationship between the independent audit firm rotation and audit reports in the companies listed on the Tehran Stock Exchange. In addition, regarding the type of audit firm rotation, the results of the null hypothesis tests implicate that there is no significant relationship between these two variables in the companies under study.

Chapter Eleven - Assessing the combined effects of multiple independent variables

Null hypothesis testing has been used increasingly over previous years. Many papers have been released critiquing the use and interpretation of these tests, particularly in the fields of medicine and other physical sciences. This study considers the use of Analysis of Variance (ANOVA) testing, a type of null hypothesis test, in sociology research papers. In this study, peer reviewed research articles have been gathered and analyzed based on their misuse and misinterpretation. Common misuses include proof of the null hypothesis and lack of further explanation of statistically significant results in relation to argument of the research. These interpretations, or lack thereof, can be problematic for future research. Potential solutions are included.

Empirical mode decomposition is a nonparametric adaptive tool that decomposes signals into a set of zero-mean modes called intrinsic mode functions (IMFs) that can be used to denoise a signal by selecting the relevant (noise-free) modes. In this paper, the statistical properties of IMFs, produced by a range of signal distributions, are examined. Insight from the statistical analysis is used in a null hypothesis test that validates a best fit model distribution of the IMFs. This test is then used in a de-noising scheme, which is evaluated using different test signals corrupted with white and colored noise. Simulations under different scenarios demonstrate the efficacy of the proposed method as compared to other EMD-based de-noising techniques.