Discussion: An estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek

Abstract

The original paper by Ioannidis (2005) has done much to raise the awareness to the possible sources of biases in current scientific work. However, Ioannidis described what may be happening in the scientific literature if scientists use blindly the marginal 0.05 significance level rule throughout their work. From this possibility, he reached the alarming conclusion that most research findings are false. In the current work, Prof. Jager and Prof. Leek took it upon themselves to find what actually is happening in current medical research if we assume that the practice is to publish only the findings significant at the 0.05 level. Since Ioannidis was actually arguing about the false discovery rate (FDR) in research (without using this terminology), they utilized FDR methods that are being used in a single research project to study the phenomenon across studies. They further developed modeling tools to address the peculiarities of the data they encountered, such as rounding to specific digits, and they developed relevant text extraction algorithms, allowing them to collect data on the entire population of important medical papers. Finally, they produced the analysis in a reproducible way, which allows everyone to repeat, check, and modify their analysis (as we did). All in all, we congratulate them for shouldering the challenge in such a way. We raise three questions. (1) Does the |$\sim 14\pm 1{\%}$| science-wise FDR reflect the situation in medical research? (2) How can we improve the estimation? (3) Whatever the actual number is, if it is well above the perceived 5% FDR, how can it be brought under control? (1) Is it |$14\pm 1{\%}$|? Unfortunately, this is not clear. There are problems with some of the decisions made as to the statements collected for the analysis: (i) only “|$p <0.05$|” and typographical variations to it, but not “|$p \le 0.05$|”, (ii) only statements regarding significance expressed as “|$p=$|” and typographical variations to it, but not confidence intervals, and (iii) only statements from the Abstracts were analyzed. Each one of these decisions affects the estimate. With regard to decision (i): repeating their analysis (using their available and accessible tools) while moving the threshold slightly to “|$p\le 0.05$|” increased their science-wise FDR estimate to 20.5%, much beyond the reported |$\sim 14 \pm 1{\%}$|⁠. With regard to decision (ii): selection of the significant findings at the 0.05 level is a very common practice even when reporting confidence intervals (CIs). It takes the form of reporting in the abstract the marginal 95% CIs that do not cover 0 (or do not cover 1 for odd ratios and relative risks). Although we did not quantify this phenomenon, many studies report only CIs and hence were entirely excluded from the analysis, with unknown effect on the estimate. More importantly, in some studies that report CIs and |$p$|-values, reported |$p$|-values tend to escort only the more decisive CIs. With regard to decision (iii): examining in depth a sample of 25 papers from those used in the original analysis, 5 from each journal, revealed that the number of significance statements in the body of the papers varied much, from 1 to 68 with a median of 24, while the median of abstract-only data in the same papers was 4, close to the value 2 reported for the entire database. The implication is that there is a strong selection-into-the-abstract effect. As further evidence, we observed that in 19 of the 25 papers that we studied in depth the smallest |$p$|-value in the paper was further selected into the abstract. One result of such a selection is that the distribution of the |$p$|-values of false discoveries in the abstracts that are smaller than 0.05 is stochastically smaller than uniform. The authors did study the implication of this problem on their estimate. From their Figure 5(b), which describes the effect of selecting the smallest of 20 uniformly distributed |$p$|-values into the abstract, one can approximate that the “true science-wise” FDR is about 30% when the estimate is 20%. While Jager and Leek discuss it as an extreme deviation from the assumptions, in view of the above we see it as a realistic or even optimistic scenario. So far we discussed the data being used. There are also some problems with the analysis. The authors treated appropriately the values of 0.04, 0.03, 0.02, and 0.01 as being the result of rounding. But then they avoided using the same approach to address the practice of “rounding on a log-base-10 scale”, even though only 7.8% of the |$p$|-values are at 0.01, while 36.9% are at 0.001, and 8.8% at 0.0001. More importantly, any |$p$|-value reported as “|$p\le x$|” was treated as left censored at |$x$|⁠. This means that if |$p\le 0.04$| is reported, it is treated as |$0.00<p \le 0.04$|⁠, and not as |$0.03

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 54 citations exist upstream.