In Defense of Hypothesis Testing: A Response to the Joint Editorial From the International Society of Physiotherapy Journal Editors on Statistical Inference Through Estimation

Abstract

In June 2022, Elkins et al1 published an editorial in PTJ on behalf of the International Society of Physiotherapy Journal Editors (ISPJE) (hereafter referred to as “the editorial”), proposing that ISPJE journals require researchers to stop using null hypothesis significance tests and adopt “estimation methods” in which the 95% CI is compared with the smallest effect of interest. In response, I posted a critical commentary as a preprint, which has since been peer-reviewed and published.2 However, I then learned that the original editorial had actually been published in the Journal of Physiotherapy in January 2022,3 and a number of other methodologists had already written critiques.4,5 Lakens’s critique4 had already been published in the Journal of Physiotherapy by the time of my preprint, with a subsequent response from Elkins et al6 (hereafter referred to as “the response”). Thus, in order to comment on the editorial in PTJ, I also need to acknowledge past critiques and the response to Lakens. 95% CIs and corresponding P values for testing |${H}_0\!:{\mu}_1-{\mu}_2=0$| (null hypothesis significance test [NHST]) and |${H}_0\!:{\mu}_1-{\mu}_2\le 1$| (1-sided minimal effects test [MET]). Open circles indicate when the NHST is statistically significant, |$\alpha =.05$|⁠. I commend my more attuned colleagues to being aware of the original editorial in January, and I will save readers some time by saying that we have a lot of overlap in our concerns.2,4,5 Primarily, I would argue that the editorial (1) does not adequately deal with the fact that P values and CIs are based on the same mathematics, such that any weaknesses of P values are shared with CIs; (2) presents several misleading statements about P values and hypothesis testing; and (3) ultimately proposes an alternative that is a hypothesis test in itself. Specifically, the editorial proposes comparing the 95% CI with the smallest effect size of interest (hereafter |$\delta$|⁠). Although the editorial proposes this comparison informally, this comparison is mathematically equivalent to a 1-sided null hypothesis test in which |${H}_0:{\mu}_1-{\mu}_2\le \delta$|2,4 (Figure, values A and B) and is referred to as a minimal or minimum effects test.7,8 Arguments in the response highlight the importance of CIs and the value of leaning into the uncertainty in our data. However, the arguments do not change the fundamental problem with the editorial: after proposing to ban hypothesis tests, the editorial ultimately proposes a minimal effects test against the smallest effect size of interest, but without explicit formulation.2,4 Thus, the central logic of the call to ban hypothesis tests is questionable, and I want to further draw attention to 3 issues in the response. The response argues that “experimenters who obtain a significant test finding cannot expect that, if an exact replication of their study were possible, it too would obtain a significant finding.” The response further quotes Amherein et al,9 saying that “it would not be very surprising for one to obtain P < .01 and the other P < .30” even with 80% power to detect a genuine effect. First, if you are studying a genuine effect with 80% statistical power in each replication, then by definition 80% of P values in the long run would fall below .05, with 20% of P values being >.05. As such, although experimenters can indeed expect the occasional P = .30 in a follow-up study, they can also reasonably expect “replication” 80% of the time (if we define replication as P < .05). Second, it is important to note that Amherein et al9 were arguing against “statistical significance” as a dichotomy to rightly prevent human errors in judgment, not arguing against hypothesis testing in general because of some mathematical flaw inherent to P values. Finally, if the P value is “bad” because it bounces around due to sampling variability, then so too is the CI, because it is equally subject to sampling variability. Compare the Figure’s values C with D (for the null hypothesis significance test) and E with F (for the minimal effects test). The response would argue that the P value here is unstable because the result goes from nonsignificant to significant in both cases. However, the 95% CI similarly goes from including a null value to excluding a null value (0 in C and D, or 1 in E and F). Thus, the editorial’s proposed alternative is no more stable than the P value. The response argues that the null is self-evidently false because no tested effect will ever be perfectly zero, arguing that “the only reason (the null is) not always found to be false is that almost all studies lack the precision to detect tiny effects.” If we accept the response’s insistence on hyperprecision (ie, that the null is false if the true effect is .001 rather than .0), then the same logic can be applied to the alternative: no effect will ever perfectly equal the smallest effect size of interest either, but the editorial was willing to assume the |$\delta$| value as a point-estimate in the proposed alternative. Everything has some degree of measurement error. It makes much more practical sense to think about |${H}_0=0$| as a model, not some infinitesimally precise prediction. To the extent that we can measure an effect, a lot of tested effects will be 0, and it should not be hard to show that a useful intervention is discernably different from 0! If we want to set an even higher standard, we could set |${H}_0\le \delta$|⁠, where |$\delta >$| 0, but then we need to define what |$\delta$| should be in a given population5,10 and determine an adequate study design. Further, the response’s argument for hyperprecision applies only to the point null of a 2-sided hypothesis test H0 = 0. The true effect does not need to be perfectly 0 for a 1-sided null to be true, H0 ≤ 0 or H0 ≥ 0, which begs the question: would the ISJPE allow P values from one-sided tests? Furthermore, testing the null hypothesis is very helpful in situations when we have multiple degrees of freedom being tested simultaneously. Consider an omnibus F-test for the group × time interaction in a trial with 3 arms and 3 time points. What would the editorial propose in this situation? One could draw many CIs comparing many groups at many times, but I would argue it is still useful to obtain a P value for the interaction first to see if there is any evidence that the effect of time depends on group before delving into specific post-hoc comparisons. The response states, “In the absence of a well-established threshold for interpretation, authors can still interpret a confidence interval by describing the practical implications of all values inside the confidence interval.” I think the second part of this statement is a good idea, and thus it is very important to report effect sizes and CIs or better yet to share your data for maximum usability.11,12 However, the CI provides the range of population parameters that are compatible with the data that we observed to a given level of uncertainty.13,14 That is, any parameter value within the 95% CI would yield P > .05 if we tested the probability of getting data as or more extreme than our current sample, assuming the null were true (ie, |$P(\ge \overline{x}|{H}_0)$|⁠), and the limits of the CI are where P would turn to <.05. Consider the Figure, values E vs F: when formalized as a minimal effects test, only F should lead to the decision that |$\delta$| is incompatible with the effect we observed in the data. If authors are allowed to informally interpret all values within the CI, as the response argues, they might focus on the upper tail of panel E, downplaying the fact that E is an “equivocal” finding and the data are compatible with many values of |${H}_0\le \delta$|⁠. History shows us that authors are overly optimistic about their data when statistical benchmarks are removed.15–17As such, I think the response assigns far too much credit to authors and readers while deflecting responsibility away from editors and reviewers. If we cannot trust the average author with P values, should we trust them with a qualitative interpretation of CIs? The solution is neither banning P values nor CIs. We need more statistical training focused on (correctly) interpreting P values, avoiding Bayesian misinterpretations of frequentist statistics, using minimal effect and equivalence tests, and so on. We also need more training on how to avoid questionable research practices and pitfalls. P values get a disproportionate amount of attention in popular conversations about methodology,18,19 but underpowered subgroup analyses, surrogate outcomes, P-hacking, hypothesizing after results are known, and selective reporting all pose far greater threats to statistical and scientific integrity.20–24 Rather than discouraging valid tools of statistical inference, I would prefer to see the ISPJE promote things like data sharing, registered reports, results-blind peer review, or even “data papers” that allow authors to archive data and receive academic credit without attempting to draw inferences from limited samples. Ultimately, I agree with the editorial that measures of effect size and CIs should be reported and interpreted in the context of the research question. I also agree we should encourage authors to lean into the uncertainty of their data and avoid misinterpretations of P values. However, estimation cannot be truly separated from hypothesis testing (or vice versa), and the editorial’s proposed ban on hypothesis tests while then proposing a minimal effects test is illogical. I would encourage the ISPJE editors to invest in making published research more transparent, reproducible, and methodologically rigorous—regardless of the method of inference. Many thanks to Dr Emma Johnson for providing critical feedback and edits on early drafts of this Point of View. There are no funders to report for this work. The author completed the ICMJE Form for Disclosure of Potential Conflicts of Interest and reported no conflicts of interest.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 7 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.