Comment: What p -hacking really looks like: A comment on Masicampo and LaLande (2012)

Daniël Lakens

56 citations0/0 files storedQuarterly Journal of Experimental Psychology

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Statistical Methods and Bayesian Inference Psychology Research and Bibliometrics

Abstract

Masicampo and Lalande (2012; M&L) assessed the distribution of 3627 exactly calculated p-values between 0.01 and 0.10 from 12 issues of three journals. The authors concluded that "The number of p-values in the psychology literature that barely meet the criterion for statistical significance (i.e., that fall just below .05) is unusually large". "Specifically, the number of p-values between .045 and .050 was higher than that predicted based on the overall distribution of p."\nThere are four factors that determine the distribution of p-values, namely the number of studies examining true effect and false effects, the power of the studies that examine true effects, the frequency of Type 1 error rates (and how they were inflated), and publication bias. Due to publication bias, we should expect a substantial drop in the frequency with which p-values above .05 appear in the literature. True effects yield a right-skewed p-curve (the higher the power, the steeper the curve, e.g., Sellke, Bayarri, & Berger, 2001). When the null-hypothesis is true the p-curve is uniformly distributed, but when the Type 1 error rate is inflated due to flexibility in the data-analysis, the p-curve could become left-skewed below pvalues of .05.\nM&L (and others, e.g., Leggett, Thomas, Loetscher, & Nicholls, 2013) model pvalues based on a single exponential curve estimation procedure that provides the best fit of p-values between .01 and .10 (see Figure 3, right pane). This is not a valid approach because p-values above and below p=.05 do not lie on a continuous curve due to publication bias. It is therefore not surprising, nor indicative of a prevalence of p-values just below .05, that their single curve doesn't fit the data very well, nor that Chi-squared tests show the residuals (especially those just below .05) are not randomly distributed.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 56 citations exist upstream.

References

2013P-curve: A key to the file-drawer.2014Registered Reports 2014Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size 2012A peculiar prevalence of p values just below .05 2013The Life of p : “Just Significant” Results are on the Rise

Cited by

2025An Aberrant Abundance of Cronbach’s Alpha Values at .70 2015On the challenges of drawing conclusions from p -values just below 0.05 2015Blinded by the Light: How a Focus on Statistical “Significance” May Causep-Value Misreporting and an Excess ofp-Values Just Below .05 in Communication Science 2022Big Little Lies: A Compendium and Simulation of p-Hacking Strategies 2017Reanalyzing Head et al. (2015): investigating the robustness of widespread p -hacking 2018Effect Size Estimation Fromt-Statistics in the Presence of Publication Bias 2023Assessing the Evidential Value of Mental Fatigue and Exercise Research 2019Research practices and statistical reporting quality in 250 economic psychology master's theses: a meta-research investigation

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.