Skip to content
Faster Than Normal
General Thinking & Meta-Models

P-hacking

Model #0734Category: General Thinking & Meta-ModelsDepth to apply:

By Updated 3 sources

4 min read

Contents

  1. 1. Core Idea
  2. 2. How to See It
  3. 3. How to Use It
  4. 4. Founders & Leaders
  5. 5. Connected Models
  6. 6. One Key Quote
  7. 7. Summary & Further Reading
General Thinking & Meta-Models
Section 1

Core Idea

P-hacking is the practice of manipulating statistical analysis — testing multiple hypotheses, adjusting variables, cherry-picking subgroups — until a result crosses the p < 0.05 significance threshold. The finding looks rigorous but is noise dressed as signal. In research, it produces non-replicable results at industrial scale. In business, the same pattern appears whenever teams torture data until it confirms a preferred narrative: A/B tests run past their stopping rule until one variant wins by chance, metrics sliced until a segment looks impressive, dashboards built to justify decisions already made. The model warns that statistical significance without methodological discipline is a mirage — and that the incentive to find "a result" will corrupt any analytical process that lacks pre-committed rules.

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Section 2

How to See It

Product
You're seeing it when a team runs an A/B test "just a little longer" after the first read disappoints, or checks multiple metrics until one shows significance. The experiment has been hacked to produce an answer.
Fundraising
You're seeing it when a pitch deck highlights a customer segment that grew 40% — after trying twelve different cuts. The impressive number is selection artefact, not signal.
Strategy
You're seeing it when leadership requests "more analysis" until the data supports the preferred direction. The conclusion was fixed; the analysis was search.
Section 3

How to Use It

Pre-register your hypotheses before running the test — in product experiments, market research, and strategic analysis. Define what you're measuring and what counts as success before you see data. When someone presents a surprising statistic, ask how many things they tested to find it. Build a culture where null results are respected, not buried, so the incentive to hack disappears.
Decision filter
"Was the hypothesis defined before data collection, or did we search until we found a result?"
As a founder
Hold your team to pre-registered hypotheses. Define success metrics before launching experiments, not after. When a team presents an unexpected insight, ask how many cuts they tried. Reward honest null results over manufactured positives.
Section 5

Founders & Leaders

Jim SimonsMathematician; founder, Renaissance Technologies
Simons built Renaissance Technologies on statistical arbitrage — and understood that most apparent patterns in financial data are noise. His team ran rigorous backtests with strict out-of-sample validation, rejecting strategies that only "worked" through data mining. The discipline was refusing to see signal where only noise existed. Founders face the same trap in smaller form every week: A/B tests nursed past their stopping rules, metrics sliced until they tell the story the board wants to hear, cohort analyses that conveniently confirm the roadmap. Simons' lesson is structural: insist on pre-registered hypotheses and out-of-sample validation before acting on any quantitative finding. The cost of p-hacking isn't a bad research paper — it's building strategy on statistical artefacts that dissolve under replication.
Section 7

Connected Models

Reinforces
Data Dredging
Data dredging is the broader practice of searching data without pre-defined hypotheses. P-hacking is its statistical endpoint — both produce "findings" that are artefacts of search, not discovery.
Reinforces
Confirmation Bias
Confirmation bias drives the analyst to keep testing until the data agrees with the prior belief. P-hacking is what confirmation bias looks like when it reaches the spreadsheet.
Tension
Reproducibility
Reproducibility is the standard p-hacking violates. If a result was found through data manipulation, it won't replicate — making reproducibility the natural test for p-hacked findings.
Section 8

One Key Quote

"If you torture the data long enough, it will confess to anything."
— Ronald Coase
Section 11

Summary & Further Reading

P-hacking is manipulating analysis until it produces a statistically significant result. It turns noise into false signal. Defend against it by pre-registering hypotheses, fixing sample sizes in advance, and rewarding null results.
01
Science Fictions — Stuart Ritchie (2020)
Book
How fraud, bias, negligence, and hype undermine the search for truth.
02
Statistical Rethinking — Richard McElreath (2020)
Book
Bayesian approach to data analysis that sidesteps p-value traps.
03
The Man Who Solved the Market — Gregory Zuckerman (2019)
Book
How Jim Simons built Renaissance Technologies on rigorous statistical discipline.

Why this matters next

mental modelsConfirmation Bias

P-hacking applied the Confirmation Bias mental model

mental modelsNarrative

P-hacking applied the Narrative mental model

mental modelsScale

P-hacking applied the Scale mental model

mental modelsReproducibility

P-hacking applied the Reproducibility mental model

mental modelsStatistical Significance

P-hacking applied the Statistical Significance mental model

mental modelsCost

P-hacking applied the Cost mental model

Frequently asked questions

What is P-hacking?

P-hacking is a mental model used for better thinking and decision-making.

How do you apply P-hacking?

To apply P-hacking, identify situations where this framework is relevant, then use it as a lens to evaluate your options and decisions. The model is most useful when combined with other complementary mental models.

What category does P-hacking fall under?

P-hacking falls under the General Thinking & Meta-Models category of mental models. Other models in this category can be found on the General Thinking & Meta-Models hub page.

Why is P-hacking important?

P-hacking is important because it provides a structured way to think about problems that would otherwise be approached with intuition alone. Understanding this model helps you avoid common reasoning errors and make better decisions.

Continue exploring

5W

Mental model

5 Whys

A root-cause analysis technique that drills past symptoms by asking why five suc

AW

Mental model

All Models Are Wrong

George Box s insight that every model is a simplification of reality — the quest

AL

Mental model

Amara's Law

We tend to overestimate the effect of a technology in the short run and underest

EM

Mental model

Eisenhower Decision Matrix

A prioritization framework that sorts tasks into four quadrants based on urgency

FT

Mental model

First Principles Thinking

Breaking down complex problems to their most fundamental truths, then reasoning

GL

Mental model

Goodhart's Law

When a measure becomes a target, it ceases to be a good measure — incentives shi

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Popular Mental Models

First Principles ThinkingOccam's RazorCircle of CompetenceInversionConfirmation BiasSecond-Order ThinkingDunning-Kruger EffectSurvivorship BiasPareto PrincipleOpportunity Cost