General Thinking & Meta-Models
Section 1
Core Idea
P-hacking is the practice of manipulating statistical analysis — testing multiple hypotheses, adjusting variables, cherry-picking subgroups — until a result crosses the p < 0.05 significance threshold. The finding looks rigorous but is noise dressed as signal. In research, it produces non-replicable results at industrial scale. In business, the same pattern appears whenever teams torture data until it confirms a preferred narrative: A/B tests run past their stopping rule until one variant wins by chance, metrics sliced until a segment looks impressive, dashboards built to justify decisions already made. The model warns that statistical significance without methodological discipline is a mirage — and that the incentive to find "a result" will corrupt any analytical process that lacks pre-committed rules.
Free playbook
Get The Business Model Playbook
58 business models, one visual page each: how the money flows, the metrics that matter, and who runs it. Free when you join the Faster Than Normal email.
Free. No spam. Unsubscribe anytime.
Section 2
How to See It
Product
You're seeing it when a team runs an A/B test "just a little longer" after the first read disappoints, or checks multiple metrics until one shows significance. The experiment has been hacked to produce an answer.
Fundraising
You're seeing it when a pitch deck highlights a customer segment that grew 40% — after trying twelve different cuts. The impressive number is selection artefact, not signal.
Strategy
You're seeing it when leadership requests "more analysis" until the data supports the preferred direction. The conclusion was fixed; the analysis was search.
Section 3
How to Use It
Pre-register your hypotheses before running the test — in product experiments, market research, and strategic analysis. Define what you're measuring and what counts as success before you see data. When someone presents a surprising statistic, ask how many things they tested to find it. Build a culture where null results are respected, not buried, so the incentive to hack disappears.
Decision filter
"Was the hypothesis defined before data collection, or did we search until we found a result?"
As a founder
Hold your team to pre-registered hypotheses. Define success metrics before launching experiments, not after. When a team presents an unexpected insight, ask how many cuts they tried. Reward honest null results over manufactured positives.
Section 5
Founders & Leaders
Jim SimonsMathematician; founder, Renaissance Technologies
Simons built Renaissance Technologies on statistical arbitrage — and understood that most apparent patterns in financial data are noise. His team ran rigorous backtests with strict out-of-sample validation, rejecting strategies that only "worked" through data mining. The discipline was refusing to see signal where only noise existed. Founders face the same trap in smaller form every week: A/B tests nursed past their stopping rules, metrics sliced until they tell the story the board wants to hear, cohort analyses that conveniently confirm the roadmap. Simons' lesson is structural: insist on pre-registered hypotheses and out-of-sample validation before acting on any quantitative finding. The cost of p-hacking isn't a bad research paper — it's building strategy on statistical artefacts that dissolve under replication.
Section 7
Connected Models
Reinforces
Data Dredging
Data dredging is the broader practice of searching data without pre-defined hypotheses. P-hacking is its statistical endpoint — both produce "findings" that are artefacts of search, not discovery.
Reinforces
Confirmation Bias
Confirmation bias drives the analyst to keep testing until the data agrees with the prior belief. P-hacking is what confirmation bias looks like when it reaches the spreadsheet.
Tension
Reproducibility
Reproducibility is the standard p-hacking violates. If a result was found through data manipulation, it won't replicate — making reproducibility the natural test for p-hacked findings.
Section 8
One Key Quote
— Ronald Coase"If you torture the data long enough, it will confess to anything."
Section 11
Summary & Further Reading
P-hacking is manipulating analysis until it produces a statistically significant result. It turns noise into false signal. Defend against it by pre-registering hypotheses, fixing sample sizes in advance, and rewarding null results.
01
Book
How fraud, bias, negligence, and hype undermine the search for truth.
02
Book
Bayesian approach to data analysis that sidesteps p-value traps.
03
Book
How Jim Simons built Renaissance Technologies on rigorous statistical discipline.