Seeking and reporting apparent research misconduct: errors and integrity

Abstract

In his recent article on ‘Data fabrication and other reasons’ for non-random sampling in randomised clinical trials in the specialty of anesthesia, Dr Carlisle claims a statistical algorithm detects fabrication and falsification 1. Briefly, the ‘Carlisle method’ 2 evaluates the baseline demographic variables (age, weight, etc.) in a clinical trials manuscript (i.e., typically a ‘Table 1’), assumes that every individual baseline variable should be randomly assigned across treatment groups, and identifies those with more or less balance than would be expected by chance to be inconsistent with random sampling, and then categorises such data as suspicious, with potential causes ranging from typographical error to fabrication or to fraud and in need of retraction 1, 3, 4. We have already published our view of the Carlisle method for the readers of Anesthesiology 5. We did so because 12 of the articles explicitly identified by Carlisle as potentially fraudulent were published in our journal and we deemed it necessary to address the journal's readership directly. We now summarise our view, for Anaesthesia readers, of the multiple defects in the Carlisle article. First, it is factually incorrect, because it identified 12 randomised clinical trials published in Anesthesiology as suspicious because of unusual data distributions, but in fact, six of those ‘randomised clinical trials’ were actually animal studies (rats, rabbits, pigs, and non-human primates), not clinical trials. Such lack of attention to detail, in an analysis that criticises the lack of attention to detail, is disappointing. Second, the Carlisle method is flawed, leading to incorrect inferences about groups being inadequately randomised. One flaw is that the very idea of using statistical inferences to formally test the balance between randomised groups has been rejected by statisticians because the balance between two randomised groups is a property of that sample, not a property of the population from which the samples were drawn 6-8. Stated simply, when examining group balance in randomised clinical trials the null hypothesis being tested is not logically sound. Another flaw is much more consequential, arising from the assumption that all variables in a ‘Table 1’ should be independent of one another. This might be correct if each variable was randomised by itself, but trials randomise people, not variables, into treatment groups. Far from being benign, to ignore that variables in ‘Table 1’ can be correlated (e.g. taller people likely weigh more) leads to massive potential bias in the expected distribution of how even well-randomised trials should be distributed. These significant methodological problems have been discussed elsewhere and in greater detail, both online (https://steamtraen.blogspot.com/2017/06/exploring-john-carlisles-bombshell.html, https://www.methodsman.com/blog/fraud-carlisle, https://errorstatistics.com/2017/07/01/s-senn-fishing-for-fakes-with-fisher-guest-post/, all last accessed 24/08/2017) and in peer-reviewed publication 5, 9. Third, owing to these flaws, the Carlisle method lacks validity to support the ‘hope that other journals will follow suit and also screen submissions’ (using the Carlisle method) 2, and for the call ‘that every journal in the world now needs to apply Carlisle's method to all the randomised clinical trials they've ever published’ 10. Fourth, the Carlisle article does a disservice to the authors of published articles it deems suspicious. Guidelines from the Committee on Publication Ethics (COPE, a forum on publication ethics for editors and publishers) state that in suspected fabricated data in a published article, the first step is to ‘contact the author to explain concerns but do not make direct accusations’ (https://publicationethics.org/resources/flowcharts). The Carlisle article, as the first action, publically identified (in an online Appendix) the authors, titles, journals, and citations of the 78 articles that it deemed suspect. Neither the journals nor the authors of these 78 articles were notified, or given a chance to respond, prior to being ‘outed’ by Carlisle's article. The Carlisle article appears to directly contravene COPE guidance. These ethical problems have been discussed in greater detail elsewhere 5, 9. We also read the accompanying editorial which celebrated the Carlisle article, the statistical method therein, the application thereof to detecting scientific misconduct, and which promoted the ubiquitous use of the Carlisle method 10. Given all of the flaws outlined above, the editorial appears misplaced. More tragically, however, is the irresponsible language in the editorial, which calls non-normally distributed data ‘corrupted’, excoriates authors of previously published articles which were deemed ‘suspicious’ in the Carlisle article, and calls out those in association as ‘miscreants’ 10. This language has no place in a scholarly publication such as Anaesthesia. While the goal of detecting scientific misconduct is laudable, and one which journal editors universally share, the deep and multiple flaws in the Carlisle’s article and the accompanying editorial require rectification. We deplore scientific misconduct, which stains the scientific enterprise. Misconduct is arguably even more egregious when it occurs in randomised clinical trials, for such research can have immediate implications for people's lives. The responsible conduct of research is the bedrock on which the scientific enterprise rests. Scientific integrity is indispensable. While typographical and other reporting errors unfortunately do occur in published research 11, this is not misconduct. A valid method to detect fabrication and falsification (akin to plagiarism checking software) would be welcome. The Carlisle method is not such.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 15 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.