치의학 연구에서 귀무가설 유의성 검정의 문제점과 대안에 관한 고찰

Hypothesis tests

IntroductionIn the last three decades, a critical movement occurring within quantitative psychological research began to develop. This development emerged in response to the misguided use of classical based on null hypothesis significance testing (NHST). promotes dichotomous thinking and provides limited information regarding the essence of investigated phenomena. Dichotomous thinking in science -- manifested as only accepting or rejecting research hypotheses -- prevents the advancement of science and may even skew the accumulation of knowledge by stimulating the exclusive publication of studies whose hypotheses have been accepted. The so-called new statistics movement critically challenges postulates and operations. This movement is an approach based on the analytical tool of estimation (Cummings, 2014), which promotes the use of effect size as descriptive statistic, confidence intervals as inferential statistics, and meta-analysis as a reliable form of knowledge accumulation. This paper is intended to analyze the disadvantages of and the advantages of using effect size, confidence intervals and statistical power in quantitative psychological research, especially in clinical studies. Also noted and stressed is the need for editors of scientific psychological journals to adhere to policies recommended by the A.P.A. in this regard.Quantitative research in psychologyTypically, quantitative psychological research is focused on detecting the occurrence of certain population phenomena by analyzing data from a sample. An example is the case in which a researcher wishes to know if a treatment to improve the quality of life of those who suffer from breast cancer performs better than a placebo treatment for another group or those on a waiting list (also known as the control group or contrast group) (Wilkinson, 1999). Similarly, to make the decision to confirm that an independent variable or treatment did or did not have an important effect, is used. In quantitative research methodology, there are two ways of quantifying this effect: (1) Null Hypothesis Significance Testing and (2) Effect Size (ES), as well as its respective confidence intervals (Cl). These two approaches are reviewed below.Null Hypothesis Significance Testing (NHST)NHST comes from the effect size on the population, the size of the sample used and the alpha level or that is selected (p being the abbreviation for probability). Most psychology research is focused on rejecting the null hypothesis and obtaining a small instead of observing the relevance of the results that are obtained (Kirk, 2001).Among the limitations of NHST, we found its sensitivity to sample size, its inability to accept the null hypothesis, and its lack of capacity to determine the practical significance of the statistical results.Kirk (2001) states that only establishes the probability of obtaining a more or less extreme effect if the null hypothesis is true. It does not, however, communicate the magnitude of the effect or its practical significance, meaning whether the effect is found to be useful or important. As a result, inferential statistical testing has been criticized; as expressed by Ivarsson, Andersen, Johnson, and Lindwall (2013): p levels may have little, if anything, to do with real-world meaning and practical value (p.97). Some authors, such as Schmidt (1996), even suggest that statistical contrast is unnecessary and recommend focusing only on ES estimation, and Cohen (1994) suggests that NHST has not only failed to support the advance of psychology as a science but also has seriously impeded it. (p. 997).Ronald Fisher was the father of modern and experimental design. Since his time, it has been established as a convention that the for statistical significance must be less than .05, which means that an observed difference between two groups has less than a 5% probability of occurring by chance or sampling error if the null hypothesis is assumed to be true initially. …

Are there benefits from NHST?

REX B. KLINE Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research Washington, DC: American Psychological Association, 2004, 336 pages (ISBN 1-59147-118-4, US$49.95 Hardcover) In 1999, a blue-ribbon task force assembled by the American Psychological Association published their findings with regards to the long-standing controversy pertaining to null hypothesis testing (NHST). The task force dictated effect sizes and confidence intervals be reported, and p values and dichotomous accept-reject decisions be given less weight. Editorial policies in a number of journals came to reflect the views of the task force as did a subsequent revision to the American Psychological Association Publication Manual. Rex B. Kline wrote Beyond Significance Testing. Reforming Data Analysis Methods in Behavioral Research as a follow-up to both the task force recommendations and the revision to the publication manual. Kline's 1998 book Principles and Practice of Structural Equation Modeling (Guilford Press) was well received and a second edition is being published this fall. In Beyond Significance Testing, Kline reviews the controversy regarding testing, offers methods for effect size and confidence interval estimation, and suggests some methodologies. There is an accompanying website that includes resources for instructors and students. Part I of the book is a review of fundamental concepts and the debate regarding testing. Part II provides statistics for effect size and confidence interval estimation for parametric and nonparametric two-group, oneway, and factorial designs. Part III examines metaanalysis, resampling, and Bayesian estimation procedures. In the first chapter, Kline provides a scholarly summary of the null hypothesis testing debate concluding with the APA task force findings and what Kline regards as ambiguous recommendations in the publication manual. Kline predicts the future will see a smaller role for traditional statistical testing (p values) in psychology. This change will take time and may not occur until the next generation of researchers are trained, but Kline anticipates the social sciences will then become more like the natural sciences in that we will report the directions and magnitudes of our effects, determine whether they replicate, and evaluate them for their theoretical, clinical, or practical significance (p. 15). Chapter 2 is a review of fundamental concepts of research design, including sampling and estimation, the logic of statistical testing, and t, F, and chi-square tests. The problems with statistical tests are revisited in Chapter 3. What follows is a long list of errors in interpretation of p values and conclusions made after null hypothesis testing. The emphasis on null hypothesis testing in psychology is also argued to inhibit advancement of the discipline. To be fair, Kline recognizes there is yet no magical alternative to statistical tests and that such tests are appropriate in some circumstances when applied correctly. Nonetheless, Kline envisions a future where effect sizes and confidence intervals are reported, substantive rather than statistical predominates, and NHST-Centric thinking has diminished. Part II covers effect size and confidence interval calculations. Chapter 4 is a presentation of parametric effect size indexes. Independent and dependent sample statistics are covered separately. The textbook's website has a supplementary chapter on twogroup multivariate designs. Group difference indexes such as d are distinguished from measures of association such as r. Case level analyses of group differences are also reviewed. Sections not relevant to a reader's needs can be skipped without loss of continuity. Interpretive guidelines for effect size magnitude and how one might be fooled by effect size estimation are sections that should not be passed over. …

Null hypothesis significance testing (NHST) is the dominant statistical approach in biology, although it has many, frequently unappreciated, problems. Most importantly, NHST does not provide us with two crucial pieces of information: (1) the magnitude of an effect of interest, and (2) the precision of the estimate of the magnitude of that effect. All biologists should be ultimately interested in biological importance, which may be assessed using the magnitude of an effect, but not its statistical significance. Therefore, we advocate presentation of measures of the magnitude of effects (i.e. effect size statistics) and their confidence intervals (CIs) in all biological journals. Combined use of an effect size and its CIs enables one to assess the relationships within data more effectively than the use of p values, regardless of statistical significance. In addition, routine presentation of effect sizes will encourage researchers to view their results in the context of previous research and facilitate the incorporation of results into future meta-analysis, which has been increasingly used as the standard method of quantitative review in biology. In this article, we extensively discuss two dimensionless (and thus standardised) classes of effect size statistics: d statistics (standardised mean difference) and r statistics (correlation coefficient), because these can be calculated from almost all study designs and also because their calculations are essential for meta-analysis. However, our focus on these standardised effect size statistics does not mean unstandardised effect size statistics (e.g. mean difference and regression coefficient) are less important. We provide potential solutions for four main technical problems researchers may encounter when calculating effect size and CIs: (1) when covariates exist, (2) when bias in estimating effect size is possible, (3) when data have non-normal error structure and/or variances, and (4) when data are non-independent. Although interpretations of effect sizes are often difficult, we provide some pointers to help researchers. This paper serves both as a beginner's instruction manual and a stimulus for changing statistical practice for the better in the biological sciences.

This paper presents a review and critique of statistical null hypothesis testing in ecological studies in general, and wildlife studies in particular, and describes an alternative. Our review of Ecology and the Journal of Wildlife Management found the use of null hypothesis testing to be pervasive. The estimated number of P-values appearing within articles of Ecology exceeded 8,000 in 1991 and has exceeded 3,000 in each year since 1984, whereas the estimated number of P-values in the Journal of Wildlife Management exceeded 8,000 in 1997 and has exceeded 3,000 in each year since 1994. We estimated that 47% (SE = 3.9%) of the P-values in the Journal of Wildlife Management lacked estimates of means or effect sizes or even the sign of the difference in means or other parameters. We find that null hypothesis testing is uninformative when no estimates of means or effect size and their precision are given. Contrary to common dogma, tests of statistical null hypotheses have relatively little utility in science and are not a fundamental aspect of the scientific method. We recommend their use be reduced in favor of more informative approaches. Towards this objective, we describe a relatively new paradigm of data analysis based on Kullback-Leibler information. This paradigm is an extension of likelihood theory and, when used correctly, avoids many of the fundamental limitations and common misuses of null hypothesis testing. Information-theoretic methods focus on providing a strength of evidence for an a priori set of alternative hypotheses, rather than a statistical test of a null hypothesis. This paradigm allows the following types of evidence for the alternative hypotheses: the rank of each hypothesis, expressed as a model; an estimate of the formal likelihood of each model, given the data; a measure of precision that incorporates model selection uncertainty; and simple methods to allow the use of the set of alternative models in making, formal inference. We provide an example of the information-theoretic approach using data on the effect of lead on survival in spectacled eider ducks (Somateria fischeri). Regardless of the analysis paradigm used, we strongly recommend inferences based on a priori considerations be clearly separated from those resulting from some form of data dredging.

P-values are misunderstood, but do not confound

Null hypothesis testing has long since been the ‘go‐to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open‐science framework, the present study (a) re‐evaluates the empirical findings of 418 t‐tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p‐values fell between .01 and .05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field‐wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.

Statistically nonsignificant (p > .05) results from a null hypothesis significance test (NHST) are often mistakenly interpreted as evidence that the null hypothesis is true-that there is "no effect" or "no difference." However, many of these results occur because the study had low statistical power to detect an effect. Power below 50% is common, in which case a result of no statistical significance is more likely to be incorrect than correct. The inference of "no effect" is not valid even if power is high. NHST assumes that the null hypothesis is true; p is the probability of the data under the assumption that there is no effect. A statistical test cannot confirm what it assumes. These incorrect statistical inferences could be eliminated if decisions based on p values were replaced by a biological evaluation of effect sizes and their confidence intervals. For a single study, the observed effect size is the best estimate of the population effect size, regardless of the p value. Unlike p values, confidence intervals provide information about the precision of the observed effect. In the biomedical and pharmacology literature, methods have been developed to evaluate whether effects are "equivalent," rather than zero, as tested with NHST. These methods could be used by biological anthropologists to evaluate the presence or absence of meaningful biological effects. Most of what appears to be known about no difference or no effect between sexes, between populations, between treatments, and other circumstances in the biological anthropology literature is based on invalid statistical inference.

Chapter Seven - The debate over null hypothesis testing and wise use as the solution

The ease with which data can be collected and analyzed via personal computer makes it potentially attractive to "peek" at the data before a target sample size is achieved. This tactic might seem appealing because data collection could be stopped early, which would save valuable resources, if a peek revealed a significant effect. Unfortunately, such data snooping comes with a cost. When the null hypothesis is true, the Type I error rate is inflated, sometimes quite substantially. If the null hypothesis is false, premature significance testing leads to inflated estimates of power and effect size. This program provides simulation results for a wide variety of premature and repeated null hypothesis testing scenarios. It gives researchers the ability to know in advance the consequences of data peeking so that appropriate corrective action can be taken.

Clinical trials targeting Alzheimer's disease (AD) aim to alleviate clinical symptoms and alter the course of this complex neurodegenerative disorder. However, the conventional approach of null hypothesis significance testing (NHST) commonly employed in such trials has inherent limitations in assessing clinical significance and capturing nuanced evidence of effectiveness on a continuous scale. In this study, we conducted a re-analysis of the phase III trial of lecanemab, a recently proposed humanized IgG1 monoclonal antibody with high affinity for Aβ soluble protofibrils, using a Bayesian approach with informed t-test priors. To achieve this, we carefully selected trial data and derived effect size estimates for the primary endpoint, the Clinical Dementia Rating Scale-Sum of Boxes (CDR-SB). Subsequently, a series of Bayes Factor analyses were performed to compare evidence supporting the null hypothesis (no treatment effect) versus the alternative hypothesis (presence of an effect). Drawing on relevant literature and the lecanemab phase III trial, we incorporated different minimal clinically important difference (MCID) values for the primary endpoint CDR-SB as prior information. Our findings, based on a standard prior, revealed anecdotal evidence favoring the null hypothesis. Additional robustness checks yielded consistent results. However, when employing informed priors, we observed varying evidence across different MCID values, ultimately indicating no support for the effectiveness of lecanemab over placebo. Our study underscores the value of Bayesian analysis in clinical trials while emphasizing the importance of incorporating MCID and effect size granularity to accurately assess treatment efficacy.

The application of null hypothesis testing to psychological research has been much criticized (e.g., Bakan, 1966; Gigerenzer & Murray, 1987). I examine the specific relevance of these and other criticisms for research on sex differences. Four specific problems are identified: (1) drawing inferences about general properties that are attributed to all members of a population; (2) the distinction between the size of p and the size and theoretical importance of a difference; (3) the frequently unjustified assumption of normality; and (4) the semantic problems inherent in the language of interpretation. A few solutions are explored, and it is concluded that descriptions of the results of sex comparisons, as well as others, must reflect the data more accurately than they now do. A number of years ago, in an invited address that I gave at the annual meeting of the Canadian Psychological Association, I criticized the way in which standard null hypothesis statistics are used in research on sex differences. I argued that in just about every known comparison of the behaviour of females and males there is generally a substantial amount of overlap between the distributions of the dependent variable for the members of each sex (if the behaviour in question is physically possible for both). This is true even if a statistical test has led to the inference that the difference is significant (p .01, p Notwithstanding such extreme cases of overlap, the mere fact that any overlap occurs at all, I reasoned, renders logically false the descriptive statements that are made when differences are said to be significant. For example, to cite a currently contentious issue, suppose it is found that the average score obtained by boys on a math test is significantly higher than that obtained by girls. Since these scores invariably overlap, (see, e.g., Benbow, 1988, p. 219; Hyde et. al., 1990), then it is false to translate the statistical result into an affirmation which states that boys are better at math than are girls. I should perhaps add that this logic also applies when girls obtain higher average scores than boys, a result which has been observed in a larger variety of situations than seems to be commonly known (Kimball, 1989).During the question period, I was asked how it is possible to make a special case that null hypothesis testing is inappropriate for sex differences, when these tests were in general use for most other kinds of psychological data. My questioner clearly did not mean to imply that significance testing should be discarded altogether (or so I understood him) but rather, since these types of statistical procedures are the standard tools of contemporary experimental psychology, and therefore must be valid, then it could hardly be legitimate to make a special case for their irrelevance to sex differences. To my embarrassment, I was unable at that time to give a satisfactory reply, neither to the questioner nor to myself. On the one hand, I was convinced that my reasoning was correct as far as sex differences are concerned, yet, on the other hand, I was still so attached to significance tests as a general tool for psychological research that I was not yet ready to think that their application could be more generally problematical.Over the intervening years I have discovered a rather substantial literature that is critical of the ways in which null hypothesis testing has been integrated into the entire psychological research process. …

Commentary: Understanding practical lot quality assurance sampling

Classical null hypothesis significance testing is limited to the rejection of the point-null hypothesis; it does not allow the interpretation of non-significant results. This leads to a bias against the null hypothesis. Herein, we discuss statistical approaches to ‘null effect’ assessment focusing on the Bayesian parameter inference (BPI). Although Bayesian methods have been theoretically elaborated and implemented in common neuroimaging software packages, they are not widely used for ‘null effect’ assessment. BPI considers the posterior probability of finding the effect within or outside the region of practical equivalence to the null value. It can be used to find both ‘activated/deactivated’ and ‘not activated’ voxels or to indicate that the obtained data are not sufficient using a single decision rule. It also allows to evaluate the data as the sample size increases and decide to stop the experiment if the obtained data are sufficient to make a confident inference. To demonstrate the advantages of using BPI for fMRI data group analysis, we compare it with classical null hypothesis significance testing on empirical data. We also use simulated data to show how BPI performs under different effect sizes, noise levels, noise distributions and sample sizes. Finally, we consider the problem of defining the region of practical equivalence for BPI and discuss possible applications of BPI in fMRI studies. To facilitate ‘null effect’ assessment for fMRI practitioners, we provide Statistical Parametric Mapping 12 based toolbox for Bayesian inference..