Effect size, confidence interval and statistical significance: a practical guide for biologists

IntroductionIn the last three decades, a critical movement occurring within quantitative psychological research began to develop. This development emerged in response to the misguided use of classical based on null hypothesis significance testing (NHST). promotes dichotomous thinking and provides limited information regarding the essence of investigated phenomena. Dichotomous thinking in science -- manifested as only accepting or rejecting research hypotheses -- prevents the advancement of science and may even skew the accumulation of knowledge by stimulating the exclusive publication of studies whose hypotheses have been accepted. The so-called new statistics movement critically challenges postulates and operations. This movement is an approach based on the analytical tool of estimation (Cummings, 2014), which promotes the use of effect size as descriptive statistic, confidence intervals as inferential statistics, and meta-analysis as a reliable form of knowledge accumulation. This paper is intended to analyze the disadvantages of and the advantages of using effect size, confidence intervals and statistical power in quantitative psychological research, especially in clinical studies. Also noted and stressed is the need for editors of scientific psychological journals to adhere to policies recommended by the A.P.A. in this regard.Quantitative research in psychologyTypically, quantitative psychological research is focused on detecting the occurrence of certain population phenomena by analyzing data from a sample. An example is the case in which a researcher wishes to know if a treatment to improve the quality of life of those who suffer from breast cancer performs better than a placebo treatment for another group or those on a waiting list (also known as the control group or contrast group) (Wilkinson, 1999). Similarly, to make the decision to confirm that an independent variable or treatment did or did not have an important effect, is used. In quantitative research methodology, there are two ways of quantifying this effect: (1) Null Hypothesis Significance Testing and (2) Effect Size (ES), as well as its respective confidence intervals (Cl). These two approaches are reviewed below.Null Hypothesis Significance Testing (NHST)NHST comes from the effect size on the population, the size of the sample used and the alpha level or that is selected (p being the abbreviation for probability). Most psychology research is focused on rejecting the null hypothesis and obtaining a small instead of observing the relevance of the results that are obtained (Kirk, 2001).Among the limitations of NHST, we found its sensitivity to sample size, its inability to accept the null hypothesis, and its lack of capacity to determine the practical significance of the statistical results.Kirk (2001) states that only establishes the probability of obtaining a more or less extreme effect if the null hypothesis is true. It does not, however, communicate the magnitude of the effect or its practical significance, meaning whether the effect is found to be useful or important. As a result, inferential statistical testing has been criticized; as expressed by Ivarsson, Andersen, Johnson, and Lindwall (2013): p levels may have little, if anything, to do with real-world meaning and practical value (p.97). Some authors, such as Schmidt (1996), even suggest that statistical contrast is unnecessary and recommend focusing only on ES estimation, and Cohen (1994) suggests that NHST has not only failed to support the advance of psychology as a science but also has seriously impeded it. (p. 997).Ronald Fisher was the father of modern and experimental design. Since his time, it has been established as a convention that the for statistical significance must be less than .05, which means that an observed difference between two groups has less than a 5% probability of occurring by chance or sampling error if the null hypothesis is assumed to be true initially. …

Are there benefits from NHST?

Statistically nonsignificant (p > .05) results from a null hypothesis significance test (NHST) are often mistakenly interpreted as evidence that the null hypothesis is true-that there is "no effect" or "no difference." However, many of these results occur because the study had low statistical power to detect an effect. Power below 50% is common, in which case a result of no statistical significance is more likely to be incorrect than correct. The inference of "no effect" is not valid even if power is high. NHST assumes that the null hypothesis is true; p is the probability of the data under the assumption that there is no effect. A statistical test cannot confirm what it assumes. These incorrect statistical inferences could be eliminated if decisions based on p values were replaced by a biological evaluation of effect sizes and their confidence intervals. For a single study, the observed effect size is the best estimate of the population effect size, regardless of the p value. Unlike p values, confidence intervals provide information about the precision of the observed effect. In the biomedical and pharmacology literature, methods have been developed to evaluate whether effects are "equivalent," rather than zero, as tested with NHST. These methods could be used by biological anthropologists to evaluate the presence or absence of meaningful biological effects. Most of what appears to be known about no difference or no effect between sexes, between populations, between treatments, and other circumstances in the biological anthropology literature is based on invalid statistical inference.

A statistically significant result, and a non-significant result may differ little, although significance status may tempt an interpretation of difference. Two studies are reported that compared interpretation of such results presented using null hypothesis significance testing (NHST), or confidence intervals (CIs). Authors of articles published in psychology, behavioral neuroscience, and medical journals were asked, via email, to interpret two fictitious studies that found similar results, one statistically significant, and the other non-significant. Responses from 330 authors varied greatly, but interpretation was generally poor, whether results were presented as CIs or using NHST. However, when interpreting CIs respondents who mentioned NHST were 60% likely to conclude, unjustifiably, the two results conflicted, whereas those who interpreted CIs without reference to NHST were 95% likely to conclude, justifiably, the two results were consistent. Findings were generally similar for all three disciplines. An email survey of academic psychologists confirmed that CIs elicit better interpretations if NHST is not invoked. Improved statistical inference can result from encouragement of meta-analytic thinking and use of CIs but, for full benefit, such highly desirable statistical reform requires also that researchers interpret CIs without recourse to NHST.

Role of Antimuscarinics Combined With Alpha-blockers in the Management of Urinary Storage Symptoms in Patients With Benign Prostatic Hyperplasia: An Updated Systematic Review and Meta-analysis.

Null hypothesis significance testing (NHST) is a common statistical analysis method in HCI. But its usage and interpretation are often misunderstood. In particular, NHST does not offer the magnitude of differences observed, which is more desirable to determine the effect of comparative studies than the p value. Effect sizes and power analysis can mitigate over-reliance on the p value, and offer researchers better informed preparation and interpretation on experiments. Many research fields now require authors to include effect sizes in NHST results, and this trend is expected to be more and more common. In this chapter, I first discuss common misunderstandings of NHST and p value, and how effect sizes can complement them. I then present methods for calculating effect sizes with examples. I also describe another closely related topic, power analysis. Power analysis can be useful for appropriately designing experiments though it is not frequently used in HCI. I present power analysis methods and discuss how they should and should not be used.

The need to compare and combine data quantitatively is becoming more frequent in studies of animal behaviour, ecology and conservation. Using a hypothetical data set, I point out some limitations of combining and comparing data using Null Hypothesis Significance Testing (NHST). First, I discuss three different aspects of data analysis that should regularly be considered: (I) effect size estimation, (2) confidence intervals estimation and, (3) power analysis. I then suggest meta-analysis as a sensible alternative method to account for some limitations of NHST. Meta-analysis is a quantitative technique for the combination and comparison of independent but similar studies. Meta-analysis allows comparison and summary of effect sizes across studies. When testing hypotheses framed in evolutionary theory, where small effects may have profound consequences, a knowledge of the magnitude of the association may be as important as knowing whether the data comply with the arbitrary, sacred and dogmatic significance criterion of p <0.05.

This paper offers a practical guide to use null hypotheses significance testing and its alternatives. The focus is on improving the quality of statistical inference in quantitative communication research. More consistent reporting of descriptive statistics, estimates of effect size, confidence intervals around effect sizes, and increasing the statistical power of tests would lead to needed improvements over current practices. Alternatives including confidence intervals, effect tests, equivalence tests, and meta-analysis are discussed.

Theory development is essential for the generation and support of research ideas. Traditional Null Hypothesis Significance Testing (NHST) has been the modus operandi for testing research questions across many branches of science since the early 20th century. The focus of a statistical test under the NHST framework considers the rejection or acceptance of a null hypothesis based on a conditional probability of the data given that the null hypothesis is true (i.e. a p-value). This approach provides no direct support for a specific theory, which often takes the form of an alternative hypothesis. Furthermore, rejection of a null hypothesis based on a p-value provides no information on the magnitude of a difference and is affected by sample size, alpha level, and effect size. Such dependency on p-values can lead to misunderstanding and misinterpretation of results and conclusions. Therefore, the limitations of NHST warrant the investigation and development of new, more rigorous approaches to theory testing. A quantitative approach, called “Testing Theory-Based Quantitative Predictions” (TTQP), has been proposed using effect size indices and confidence intervals to directly test predictions posited by theory (Velicer et al, 2008). Effect size indices provide information regarding the magnitude and direction of an effect while confidence intervals provide a means of “testing” specific predictions. This approach is an iterative process, allowing the researcher to tailor the theory as empirical data is collected. The use of the TTQP approach contributes to the movement away from NHST and the reliance on p-values, while promoting a stronger and more informative method. A quantitative orientation represents an essential change in thinking about theory testing by emphasizing the numeric strength of a measure, leading researchers away from a simple binary accept/reject framework. Predictions are made relative to the specific measure/variable, but the use of effect sizes allows for comparison across studies and across theories. Therefore, the TTQP approach actually provides more information than traditional NHST. The TTQP approach involves several steps. First, verbal descriptions of the expected values are designated a priori. These predictions are theory-based and guided by previous empirical findings (e.g. “small effect”). Second, verbal predictions are translated into quantitative values based on traditional guidelines or empirical results (e.g. “0.01”). Then, observed effect size estimates with surrounding confidence intervals are generated from sample data. If a confidence interval contains the predicted value, the prediction is confirmed. If the predicted value falls outside of the confidence interval, the prediction is

We examined the degree of autocorrelation among single‐case design data with six measures used to estimate treatment effect size. The most commonly used measures of effect size for single‐case data over the last 5 years published in peer‐reviewed journals were selected for comparison. The overall mean degree of autocorrelation was 0.46 (SD = 0.33) across the 304 data paths, which represents a moderate degree of autocorrelation. Overall, it appears that non‐parametric measures of effect size (i.e., percent of non‐overlapping data [PND], non‐overlap of all pairs [NAP], and improvement rate difference [IRD] values) were substantially and significantly more influenced by the degree of autocorrelation. Tau‐U effect size estimate was the non‐parametric exception as it was not significantly influenced by the degree of autocorrelation. Parametric measures of effect sizes such as standardized mean difference (SMD) and log response ratio (LRR) values did not appear to be significantly influenced by the degree of autocorrelation. For SMD, LRR, and Tau‐U values, the correlation between the effect size value and the degree of autocorrelation was minimal. For NAP, IRD, and PND values, the correlation between the effect size value and the degree of autocorrelation was moderate, indicating that these estimates of effect size should be avoided as the degree of autocorrelation between data points increases.

REX B. KLINE Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research Washington, DC: American Psychological Association, 2004, 336 pages (ISBN 1-59147-118-4, US$49.95 Hardcover) In 1999, a blue-ribbon task force assembled by the American Psychological Association published their findings with regards to the long-standing controversy pertaining to null hypothesis testing (NHST). The task force dictated effect sizes and confidence intervals be reported, and p values and dichotomous accept-reject decisions be given less weight. Editorial policies in a number of journals came to reflect the views of the task force as did a subsequent revision to the American Psychological Association Publication Manual. Rex B. Kline wrote Beyond Significance Testing. Reforming Data Analysis Methods in Behavioral Research as a follow-up to both the task force recommendations and the revision to the publication manual. Kline's 1998 book Principles and Practice of Structural Equation Modeling (Guilford Press) was well received and a second edition is being published this fall. In Beyond Significance Testing, Kline reviews the controversy regarding testing, offers methods for effect size and confidence interval estimation, and suggests some methodologies. There is an accompanying website that includes resources for instructors and students. Part I of the book is a review of fundamental concepts and the debate regarding testing. Part II provides statistics for effect size and confidence interval estimation for parametric and nonparametric two-group, oneway, and factorial designs. Part III examines metaanalysis, resampling, and Bayesian estimation procedures. In the first chapter, Kline provides a scholarly summary of the null hypothesis testing debate concluding with the APA task force findings and what Kline regards as ambiguous recommendations in the publication manual. Kline predicts the future will see a smaller role for traditional statistical testing (p values) in psychology. This change will take time and may not occur until the next generation of researchers are trained, but Kline anticipates the social sciences will then become more like the natural sciences in that we will report the directions and magnitudes of our effects, determine whether they replicate, and evaluate them for their theoretical, clinical, or practical significance (p. 15). Chapter 2 is a review of fundamental concepts of research design, including sampling and estimation, the logic of statistical testing, and t, F, and chi-square tests. The problems with statistical tests are revisited in Chapter 3. What follows is a long list of errors in interpretation of p values and conclusions made after null hypothesis testing. The emphasis on null hypothesis testing in psychology is also argued to inhibit advancement of the discipline. To be fair, Kline recognizes there is yet no magical alternative to statistical tests and that such tests are appropriate in some circumstances when applied correctly. Nonetheless, Kline envisions a future where effect sizes and confidence intervals are reported, substantive rather than statistical predominates, and NHST-Centric thinking has diminished. Part II covers effect size and confidence interval calculations. Chapter 4 is a presentation of parametric effect size indexes. Independent and dependent sample statistics are covered separately. The textbook's website has a supplementary chapter on twogroup multivariate designs. Group difference indexes such as d are distinguished from measures of association such as r. Case level analyses of group differences are also reviewed. Sections not relevant to a reader's needs can be skipped without loss of continuity. Interpretive guidelines for effect size magnitude and how one might be fooled by effect size estimation are sections that should not be passed over. …

Association between circulating adipocytokine concentrations and microvascular complications in patients with type 2 diabetes mellitus: A systematic review and meta-analysis of controlled cross-sectional studies

Effectiveness of psychosocial interventions for reducing parental substance misuse.

The aim of this systematic review and meta-analysis was to analyze data from randomized controlled trials (RCTs) assessing the effects of oleic acid (OA) supplementation on blood inflammatory markers in adults. PubMed, EMBASE and Cochrane Library databases were systematically searched from 1950 to 2019, with adults and a minimum intervention duration of 4 weeks. The effect size was estimated, adopting standardized mean difference (SMD) and 95% confidence interval (CI). Of the 719 identified studies, thirty-one RCTs involving 1634 subjects were eligible. The results of this study revealed that increasing OA supplementation significantly reduced C-reactive protein (CRP) (SMD: −0.11, 95% CI: −0.21, −0.01, P = 0.038). However, dietary OA consumption did not significantly affect tumor necrosis factor (TNF) (SMD: −0.05, 95% CI: −0.19, 0.10, P = 0.534), interleukin 6 (IL-6) (SMD: 0.01, 95% CI: −0.10, 0.13, P = 0.849), fibrinogen (SMD: 0.08, 95% CI: −0.16, 0.31, P = 0.520), plasminogen activator inhibitor type 1 (PAI-1) activity (SMD: −0.11, 95% CI: −0.34, 0.12, P = 0.355), soluble intercellular adhesion molecule-1 (sICAM-1) (SMD: −0.06, 95% CI: −0.26, 0.13, P = 0.595) or soluble vascular cell adhesion molecule-1 (sVCAM-1) (SMD: −0.04, 95% CI: −0.27, 0.18, P = 0.701). Overall, the meta-analysis demonstrated that dietary OA supplementation significantly reduced CRP, yet did not affect other inflammatory markers including TNF, IL-6, fibrinogen, PAI-1 activity, sICAM-1or sVCAM-1.

Heterogeneity across studies makes it difficult to draw firm conclusions regarding the effectiveness of psychosocial interventions for this population. There is an immediate need for rigorous trials with process evaluations and clearer, detailed intervention descriptions. Cost-effectiveness studies should be conducted alongside future trials.