Science is not a signal detection problem

Brent M. Wilson · Christine R. Harris · John Timothy Wixted

56 citations0/0 files storedProceedings of the National Academy of Sciences

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Academic Publishing and Open Access Psychology Research and Bibliometrics

Abstract

The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, because continuously distributed effects selected using a P < 0.05 filter must be inflated, the fact that they are smaller when replicated (reflecting regression to the mean) is no reason to sound the alarm. Considered from this perspective, recent replication efforts suggest that most published P < 0.05 scientific findings are “true” (i.e., in the correct direction), with observed effect sizes that are inflated to varying degrees. We propose that original science is a screening process, one that adopts NHST logic as a useful fiction for selecting true effects that are potentially large enough to be of interest to other scientists. Unlike original science, replication science seeks to precisely measure the underlying effect size associated with an experimental protocol via large- N direct replication, without regard for statistical significance. Registered reports are well suited to (often resource-intensive) direct replications, which should focus on influential findings and be published regardless of outcome. Conceptual replications play an important but separate role in validating theories. However, because they are part of NHST-based original science, conceptual replications cannot serve as the field’s self-correction mechanism. Only direct replications can do that.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 56 citations exist upstream.

References

2005Why Most Published Research Findings Are False 2015Estimating the reproducibility of psychological science 2013Power failure: why small sample size undermines the reliability of neuroscience 1959The Logic of Scientific Discovery.2011False-Positive Psychology 2005Why Most Published Research Findings Are False 1994The earth is round (p < .05).2017A manifesto for reproducible science

Cited by

2021A Signal Detection Approach to Understanding the Identification of Fake News 2023RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable 2020High Replicability of Newly-Discovered Social-behavioral Findings is Achievable 2024Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power 2020Consideration of Sample Size in Neuroscience Studies 2020Changing the logic of replication: A case from infant studies

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.