Magnitude-Based Inference is Not Bayesian and is Not a Valid Method of Inference.
When people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together. 1 Isaac Asimov Recently, Diong2 wrote a commentary for the Scandinavian Journal of Medicine and Science in Sports critiquing a study by Pamboris et al3 for not correctly interpreting confidence intervals that contained zero. Pamboris et al4 wrote a response letter in return, seeking to rebut Diong's critique and arguing that they were appropriately implementing Magnitude-based Inference (MBI), which they argue is "based on Bayesian inference and [the] conclusions are robust."4 They neglect, however, that multiple statisticians5-7 including Bayesian statisticians8, 9 have strongly critiqued and even called for rejection of MBI as a method of inference. In this commentary, we hope to resolve some of that ambiguity and concisely explain why MBI is not a robust method of statistical inference. In their response letter, Pamboris et al4 correctly note that Diong2 defined confidence intervals inaccurately. However, Diong's definitional error does not undermine her central argument, which is that the original authors have reached conclusions that are not justified by the data. Pamboris et al compared slow dynamic stretching (SDS) to fast dynamic stretching (FDS) in an 18-person crossover design.3 They concluded that "SDS showed greater improvement than FDS in both neuromechanical and sensorimotor performance," but the differences observed between conditions in this small sample were completely consistent with random variation due to sampling variability, as we will show. As with numerous papers in the sport and exercise literature, the use of MBI led the authors to reach overly optimistic conclusions. Pamboris et al4 claim that their conclusions are justified because the "interpretation of our results was according to the recommended MBI procedures," but MBI is not an established statistical method. To date, MBI has never been published in a statistical journal10-12 nor have equations with formal notation ever been provided. Instead, MBI is implemented in Microsoft Excel spreadsheets, solely available from www.sportsci.org, with no clear documentation or version control.13, 14 Pamboris et al4 also fail to cite the numerous critiques of MBI from statisticians in the academic literature5-9 and in the popular press.15, 16 Further, their arguments neglect that MBI has been questioned or outright banned by major sports medicine journals.17, 18 Prima facie, this evidence should make one skeptical about the legitimacy of MBI. Using the Pamboris et al paper3 as a motivating example, we will give a brief review of MBI and explain why it is not a robust method of inference. As a method, MBI arose from an understandable desire to put more emphasis on effect sizes and estimation. MBI classifies results into three categories of effect size: harmful (or negative), trivial, and beneficial (or positive). Pamboris et al3 defined as trivial any values between −0.2 and +0.2 in standardized effect size units, the default values recommended by MBI's creators,10, 11 and, in our experience, the values selected by the vast majority of MBI practitioners. Yet, this is exactly how MBI interprets it. MBI concludes that there is a 10% chance that the intervention is not beneficial and a 90% chance that it is beneficial. Interpreting P-values in this way is equivalent to interpreting frequentist confidence intervals as if they were Bayesian credible intervals; we will later show why this is incorrect. Interestingly, Pamboris et al appear to be unaware of the math underlying the spreadsheets as they state in their response letter4 that their inferences "have nothing whatsoever to do with NHST [null hypothesis significance testing]." In fact, the MBI probabilities are calculated based on standard hypothesis tests. The spreadsheets used by Pamboris et al implement two one-sided t tests with non-zero null values.5 This was shown mathematically by Welsh & Knight.5 To confirm this, we simulated data from a randomized trial with n = 10 per group and a normally distributed outcome (with SD 1.0 and means of 0.1 and 0 in the experimental and control groups, respectively). When we analyzed these data in the MBI spreadsheet for comparing two means (xCompare2groups.xls), we got a probability of benefit of 35.603%. When we ran Welch's t test for unequal variances with H0 = 0.2 and one-sided (for benefit) in SAS version 9.4, we got a P-value of 0.64397. 1-0.64397 = 0.35603. Magnitude-based Inference provides the following qualitative descriptors for the probabilities: 0.5% is most unlikely; 0.5%-5% is very unlikely; 5%-25% is unlikely; 25%-75% is possible; 75%-95% is likely; 95%-99.5% is very likely; and 99.5% is most likely.10, 11 MBI also deems effects unclear when the probabilities of harm and benefit are both elevated, typically ≥5%, which is equivalent to the 90% confidence interval overlapping both the harmful and beneficial ranges. Accordingly, in their tables, Pamboris et al3 label effects as "unlikely," "possible," "likely," "very likely," and "unclear." Statisticians have noted several problems with MBI, including that it lacks a sound mathematical foundation, incorrectly interprets frequentist statistics (P-values and confidence intervals), and gives overly optimistic inferences, increasing the risk of finding spurious effects, especially when using small samples. Magnitude-based Inference was first introduced in the peer-reviewed literature in 2006.10 The method was later criticized in the same journal by the statisticians Richard Barker and Matthew Schofield, who argued that the only way to make probabilistic statements about true effects—such as: "There is a 90% chance that the intervention is beneficial"—is to adopt a fully Bayesian approach.8 They also noted that MBI draws incorrect, and overly optimistic, inferences from confidence intervals, as shown in Figure 1. For example, in the Pamboris study,3 participants experienced an average 39.8% improvement in muscle strain in the SDS compared with the FDS condition with 90% confidence interval of −16.9% to 96.5% (Figure 1). According to MBI methodology, this gives the conclusion that SDS is "likely beneficial" compared with FDS. But the correct interpretation of this confidence interval (assuming that −16.9% is within the trivial range) is that SDS is "not harmful" relative to FDS (Figure 1). As such, concluding that "SDS showed greater improvement than FDS in both neuromechanical and sensorimotor performance" is not warranted. In 2015, another pair of statisticians—Alan Welsh and Emma Knight—again raised the alarm about MBI.5 The mathematical formulas implemented in the MBI spreadsheets are not published anywhere, so Welsh and Knight painstakingly extracted the algorithms from the cells of the Excel spreadsheets. They showed that, among other issues, the method leads to inflated rates of Type I error. Recall that Type I errors are false-positive errors. In response to this critique, MBI's creators published a 2016 paper in the journal Sports Medicine claiming that MBI has better Type I and Type II error rates than standard hypothesis testing.19 This claim is dubious at face value, since Type I and Type II error necessarily trade off. Indeed, statistician Kristin Sainani (co-author of this paper) used both math and simulation to definitively disprove their claim.6 She showed that, for typical usage, MBI yields unacceptably high Type I error rates and that MBI's sample size calculators are tuned to maximize Type I error. The crux of Welsh and Knight's, and Sainani's arguments is that MBI inflates the risk of concluding benefit for ineffective interventions. Like statistical significance testing, MBI provides thresholds of evidence. Instead of highlighting P < 0.05 or P < 0.01, MBI practitioners highlight "possible" or "likely" effects. For example, Pamboris et al3 call out one "likely" and several "possible" differences between SDS and FDS (see their table 3)3 and use these findings to justify their conclusion that "SDS showed greater improvement than FDS in both neuromechanical and sensorimotor performance." But MBI's evidentiary thresholds are extremely weak.20 Using math and simulation, we estimate that—for the way MBI is typically applied in the literature—the "possible" bar is comparable to using a significance threshold as high as P < 0.60, and the "likely" bar is comparable to using a significance threshold as high as P < 0.25.1 This means that when an intervention is completely ineffective, MBI practitioners will find false-positive "possible" effects as much as 60% of the time and false-positive "likely" effects as much as 25% of the time. Unfortunately, MBI practitioners have not been informed that these levels of evidence are weak: Pamboris et al, for instance, insist that "our conclusions are robust."4 In fact, the evidence that Pamboris et al3 present to support their claims is weak. Though Pamboris et al3 did not present standard P-values for their comparisons, it is easy to calculate these from the confidence intervals provided. The P-value gives important information about the level of evidence that the data provide regardless of whether one performs a binary significance test. For example, Pamboris et al3 found that SDS has a "likely beneficial" effect on muscle strain compared with FDS. We back-calculated the two-sided P-value for this comparison (for the null hypothesis of true effect = 0) as P(T17≥(39.8%/32.4%) or T17≤(−39.8%/32.4%)) = 0.24.2 This means that if there was truly no difference between the conditions, we would have seen a difference of 39.8% or bigger in nearly a quarter of studies just by chance. Note that all other between-condition comparisons (eg, torque) were associated with even higher P-values, meaning they represent even weaker levels of evidence. Claims from MBI studies are being propagated in the literature as definitive findings. In their discussion section, Pamboris et al3 cite a previous study21 that they say: "found a decrease in muscle fascicle strain combined with an increase in muscle stiffness." This reference is to an MBI study that provided similarly weak levels of evidence for its conclusion. Yet, the findings are not qualified as being based on weak evidence, but instead are presented as conclusive. When these claims are wrong, they are false-positive errors. We advise researchers to describe their inferences about magnitudes as the legitimate reference Bayes with a dispersed uniform prior when submitting manuscripts to any journal banning MBI.22 Pamboris and colleagues' response letter adopts this tactic,4 claiming "Our inferences are Bayesian" despite never describing the methods as Bayesian in the original paper.3 In the sections that follow, we will explain what a Bayesian analysis is and why MBI is not a valid Bayesian analysis. Bayesian analysis of data does not calculate a single probability alone, but instead calculates a full probability distribution. Using the posterior probability distribution, Bayesians can compute "credible intervals" which give the probability that the true effect falls within a certain range. For example, a 90% Bayesian credible interval of −1.5 to 5.0 tells us "There is a 90% chance the true effect lies between −1.5 and 5.0." Interpreting a P-value as a Bayesian posterior probability is equivalent to interpreting a confidence interval as a Bayesian credible interval. Finally, it is important to note that just because a posterior distribution can be calculated that does not make it necessarily valid or reliable. The prior distribution represents an assumption about the nature of reality. As with any model assumption, the prior distribution must be explained, justified, and tested. The MBI spreadsheets make inferences solely based on frequentist statistics.5 They do not implement any type of Bayesian analysis. What MBI does is to interpret frequentist statistics as if they were Bayesian statistics. In this section, we will explain why this is not an acceptable approach. A defining feature of a Bayesian analysis is the incorporation of a prior probability distribution. As previously mentioned, frequentist statistics give probabilities about data given a hypothesis (eg, "The probability of the data if the intervention were equivalent to control is 10%."). The problem is that we would prefer to obtain the probability of a hypothesis given the data (eg, "There is a 90% chance that the intervention is beneficial."). Bayesian statistical methods calculate these probabilities by incorporating additional information beyond what the data alone provide. This additional information is called the "prior probability distribution." As described above, the choice of prior can greatly affect the results, particularly when dealing with small samples. Pamboris and colleagues equate MBI to "Bayesian inference with a default prior."4 But "default prior" is not sufficient detail for a scientific paper. For example, a common "default" choice when comparing means is the Cauchy probability distribution (which looks like a normal distribution but has more area in the tails). Even then, there is not a single Cauchy distribution—there is an infinite number determined by the distribution's scale parameter. A Bayesian employing a Cauchy prior would perform a sensitivity analysis to determine how the choice of scale parameter affects the results. MBI practitioners do not specify a prior and do not reflect on how the choice of prior might affect their results. This is because they do not actually incorporate a prior. Magnitude-based Inference attempts to simply bypass the prior. Rather than using the central Bayesian formula: posterior ∝ likelihood × prior, MBI instead assumes that posterior = likelihood in all cases, and thus that frequentist P-values and confidence intervals can be given a direct Bayesian interpretation. In MBI, P-values are interpreted as Bayesian posterior probabilities or, equivalently, confidence intervals are interpreted as Bayesian credible intervals. There are several problems with this: (1) MBI users are never required to specify or justify their prior, think about its implications, or perform sensitivity analyses to gauge the effect of the choice on their results; and (2) For simple problems, such as comparing means with a known standard deviation, confidence intervals are equivalent to credible intervals (and P-values to posterior probabilities) when one assumes a "flat" (or "uniform") prior.8, 23 But flat priors are unrealistic for most applications in biomedicine and sports science, as we will explain below. Magnitude-based Inference has been described as a Bayesian analysis with a flat prior,10 but writings on MBI fail to inform users of the implications of a flat prior. A flat prior means that literally all effect sizes are equally likely. This is an unrealistic assumption in biomedicine because it implies that almost every intervention has a substantial effect, many have impossibly large effects, and all of these effects are equally likely.23-25 A flat prior will almost always lead to overly optimistic inferences because—by starting with the assumption that the intervention almost certainly works (has a non-trivial effect)—it stacks the deck in favor of finding a non-trivial effect. As Zwet23 writes: "The uniform prior is not realistic at all in the context of bio-medical research. Consequently, its use leads to overestimation of the magnitude of the regression coefficient and overconfidence about its sign." As Greenland24 states: "…a flat prior is generally nonsensical in scientific terms and suboptimal for both frequentist and Bayesian decision and inference; at most flat priors only serve to bound results from optimized or sensible priors." For instance, Pamboris et al3 report that stretching increased isometric torque, a measure of muscle force, from 91 to 95 Newton meters (Nm). A flat prior implies that it is just as likely that stretching would reduce torque to zero, or conversely, create superhuman strength with post-stretching torque being 1000 Nm. This is obviously a completely unrealistic assumption. Flat priors are particularly unrealistic for sports science as empirical data show that effect sizes in sport and exercise science are predominantly small.26-28 In fact, MBI is specifically advertised as a tool that is needed because small effects predominate in sports science.22 Greenland has demonstrated that frequentist P-values and confidence intervals can be viewed as providing bounds on Bayesian posterior probabilities when assuming a prior symmetric around the null.25 We indeed believe that MBI probabilities provide an upper limit for a Bayesian posterior probability. In other words, when MBI returns "50% chance of beneficial" what this actually means is that 50% is the maximum chance that the intervention is beneficial. The choice of any other prior (assuming priors symmetric around the null) will lead to a lower probability. This means that using MBI as currently implemented will almost always present an overly optimistic picture of an intervention's effectiveness. It is easy to see that interpreting all frequentist confidence intervals as Bayesian credible intervals (or, equivalently, interpreting all P-values as Bayesian posterior probabilities) will lead to overly optimistic inferences. Just imagine if all the 95% confidence intervals in the medical literature were suddenly interpreted as Bayesian credible intervals (as MBI would allow you to do29-31!). This would imply that at least 97.5% of all significant beneficial results would represent true positives, which runs directly counter to what we know to be true based on the current reproducibility crisis in biomedicine.32-35 Finally, it is important to recognize that MBI's evidentiary thresholds (eg, "likely" and "possible") represent weak levels of evidence whether taking a frequentist or Bayesian perspective. "Likely" beneficial effects typically correspond to what Bayesians would consider "anecdotal evidence" of an effect.36 We have used the pre-post parallel groups trial spreadsheet, as this calculation tool compares the difference from pre- to post-changes between the slow and fast dynamic stretching condition thus eliminating type I error4 (emphasis added). This statement is erroneous—Type I error is not eliminated by comparing pre- and post-changes. Also, their reference to MBI as a "Bayesian analysis with a default prior" without any of its implications a of of Bayesian statistics. Finally, their claim that their inferences "have nothing whatsoever to do with a of that the Excel spreadsheets implement standard hypothesis tests. MBI this of instead the of Rather than the of their statistical researchers instead results the of the MBI spreadsheets. MBI is not alone in as a many allow users to implement statistical methods the users do not but MBI lacks review and As we have mentioned, MBI's authors have never presented a mathematical of the method and the method has never been published in a statistical MBI's documentation is also the the within the spreadsheets is at and and to following the and to one spreadsheet is This can be with other statistical and documentation are and published in the literature (eg, the Journal of of this is in the of In numerous MBI we have that authors their data with given other than that they did so to with the of This is not statistical at a that it is used on and then almost in MBI papers in the sports medicine In most cases, there is no that authors have actually their data or such as and of variances to a it is to it should not be or without can make the results to interpret (eg, what would it to that a certain type of the of time by of can also make it to simple thus it for errors and to Pamboris et al3 all but two of their with no for these The their extremely to and we believe has introduced in the inferences. For example, muscle strain by in the SDS condition (see their table and in the FDS condition (see their table due to the to a between-condition difference of (see their table This is a standardized effect size of with no of how the standardized effect size was the confidence interval for the yields inferences than the confidence interval for the effect The back-calculated P-value from the is but from the is we can use the effect size confidence interval to directly calculate the MBI beneficial probability as = But this does not the given in the which is Finally, it is unclear as to why the confidence intervals are if confidence intervals were on the data and then one would the confidence intervals to be that use the MBI Excel spreadsheets neglect to provide of their of their We note that in the of their Pamboris et al3 simply state which spreadsheets they used and then cite a that only provides for how to use the spreadsheets, not the mathematical that nor the that the This it to or the results. For example, we determine whether Pamboris et al3 used the correct analysis for comparing the SDS and FDS When comparing conditions in a crossover one to for the Pamboris et al3 they used the parallel groups trial for this but and the only for is unclear whether it correctly Indeed, we that the MBI used as a statistical error described in any statistics analysis use of the spreadsheet will in the methods of manuscripts to to more statistical (emphasis added). There is a MBI should be described in terms are for We believe it that of methods is for all scientific for and especially so when the study authors are using and are we as to the of their that sufficient is to of any statistical to the for an and nothing to the reproducibility problems in scientific Magnitude-based Inference has should to statistical significance and more to effect sizes and The is in the however, and MBI does not the problems associated with statistical significance For the mathematical and above, MBI the number of false-positive findings in the literature, does not have a and does all of these with a of Bayesian from being a MBI actually MBI has popular in exercise and sports science, because we have provided the and the provide the researchers with an for previously effects from small (emphasis added). The problem with small studies is that chance variation has a greater relative effect on it more to effects from Indeed, it is demonstrated that we should have in the results of a small study of whether the results are to or not the null MBI the risk of as it is that of MBI specifically this as an of their method in the of small samples. methods used in medical studies are first in the literature and are to and For instance, see the legitimate about significance and its use in of In some cases, statistical methods are that are later found to be by other and their use is then MBI, a of methods but this is not the MBI has never been published in a statistical journal its and be and by in methods for the of statistical Instead, papers on MBI have only been published in sports medicine or on the This can also be seen in how critiques of are with that fail to directly the raised or math and never in or to the method. Instead, of MBI have the example, first claiming that MBI has Type I and Type II and then claiming that Type I and Type II errors are because MBI is a Bayesian We have also been how MBI have found more typically in and than in For instance, in a that not a qualified that and critiques of MBI using such as of or and of with any of MBI, that the of and that this is based on the and are the most researchers in exercise and their of the inference literature is beyond This is are and using their people who can be every statistician who has MBI has concluded it to be but (with a of that just how To their the creators and of MBI have a for statistical in sport and exercise science and have legitimate on statistical of the null and studies are all However, correct of a problem does not provide a and there are of means for these problems using valid statistical of the method of we also to more errors in experimental data data and statistical and We better statistical so that researchers the and of valid analysis We also a of statistical statistics is its with for should not MBI's claim of a consistent with a "default Flat priors represent very about the nature of and are wrong for sports if one is to adopt a prior, it to be in a Bayesian analysis with and sensitivity MBI the of a prior completely and is thus definitively not Magnitude-based Inference has certain with Bayesian statistics. But you a to and describe them as just because they to a with a The claim that MBI is a Bayesian analysis is similarly just not and does This claim has led Pamboris et al4 to that, be that our analysis based on Bayesian inference methods and our conclusions are As we have shown however, conclusions from "possible" or "likely" results in MBI are Bayesian nor MBI is not a valid nor method of statistical inference and should not be This was and equally to the with being determined by a random number in
Read more