Magnitude-based Inference: Good Idea but Flawed Approach

Abstract

The statistical community has long favored the estimation of scientific importance, typically by confidence interval, over mere statistical significance, the ubiquitous P value (see Curran-Everett et al. [1,2]). The reason for this is simple: a confidence interval focuses attention on the scientific importance of some experimental result. In contrast to hypothesis tests and P values, confidence intervals are a relatively recent development: Neyman derived them in the 1930s, and medical journals have advocated their use since the 1980s (see Curran-Everett [1]). In 2006, Batterham and Hopkins (3) discussed the value of confidence intervals but then proceeded to argue their deficiencies:“In this invited commentary we first critique the traditional approach to inferential statistics, the null hypothesis test. Next we explain confidence limits, which have begun to appear in publications in response to a growing awareness that the null hypothesis test fails to deal with the real-world significance of an outcome. We then show that confidence limits alone also fail and then outline our own approach and other approaches to making inferences based on meaningful magnitudes.” Their own approach, magnitude-based inference, assigned probabilities to the interpretation of a traditional confidence interval and resulted in a probabilistic estimate that some true effect was harmful, trivial, or beneficial (3). In the guidelines published in Medicine & Science in Sports & Exercise (4), Hopkins et al. echoed the importance of the estimation of scientific importance. They also wrote, however, that they believed one outcome of the guidelines, guidelines that included a summary of magnitude-based inference, would be to“… help legitimize the innovative or controversial approaches that we and others sometimes have difficulty including in publications.”It seems to me that the proven validity of an approach is of far greater importance than the ability to cite that approach simply because it was included—without critical assessment—in some publication. In this issue of Medicine & Science in Sports & Exercise, Sainani (5) has used the algorithms written by Hopkins and Batterham (6) to demonstrate that magnitude-based inference fails to perform as its proponents claim it does. Although Sainani’s paper relies on mathematics to prove conclusively that the error rates and sample size calculations embedded in magnitude-based inference are flawed, the paper also summarizes its statistical arguments with a verbal description of the statistical errors and practical consequences of using magnitude-based inference. The implications of Sainani’s thoughtful analysis should alarm any user of magnitude-based inference. In addition to the obvious statistical errors in the magnitude-based inference approach, errors first identified by Welsh and Knight (7) and now hammered home by Sainani (5), several other aspects of magnitude-based inference are troubling: the confusing commandeering of traditional statistical terms such as type I and type II* error rates, the near-exclusive use of magnitude-based inference within the exercise biology and sports science communities, and the fact that magnitude-based inference is not advocated or even recognized within the mainstream statistical community. Moreover, Hopkins and Batterham sometimes respond to criticism of magnitude-based inference in a manner that is dismissive and without substance. For example, in their response (8) to the critique by Welsh and Knight (7), Batterham and Hopkins (8) conclude“… Welsh and Knight (8) proposed that we either present confidence intervals or use “a fully Bayesian analysis” in place of [magnitude-based inference]. However, they were silent on how to make inferences with confidence intervals…”How is the absence of guidance on how to interpret a confidence interval—many papers and textbooks do this—a sufficient counterargument to the objective statistical arguments of Welsh and Knight? Perhaps more troubling is this statement from their 2016 paper (6), a statement on which Sainani comments in her paper:“We also provide published evidence of the sound theoretical basis of [magnitude-based inference] [10, 11, 16]…”As Sainani observes, none of those three references provide anything close to evidence of the sound theoretical basis of magnitude-based inference. As the paper by Sainani (5) has confirmed, magnitude-based inference is flawed: its use results in erroneous sample sizes and an exaggerated number of false positives. Magnitude-based inference should be abandoned. As Welsh and Knight (7) and now Sainani (5) remind us, the established statistical approaches of confidence intervals and Bayesian methods are sufficient to help researchers focus on the magnitude of some scientific effect.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 11 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.