Skip to content
Faster Than Normal
Mathematics & Probability

P-values

Model #0785Category: Mathematics & ProbabilityDepth to apply:

By Updated 3 sources

5 min read

Contents

  1. 1. Core Idea
  2. 2. How to See It
  3. 3. How to Use It
  4. 4. Founders & Leaders
  5. 5. Connected Models
  6. 6. One Key Quote
  7. 7. Summary & Further Reading
Mathematics & Probability
Section 1

Core Idea

A p-value is the probability of observing data at least as extreme as the actual result, assuming the null hypothesis is true. It does not tell you the probability that your hypothesis is correct. This distinction — universally misunderstood — is the source of enormous decision-making error. A p-value of 0.04 means: "If there were truly no effect, we'd see data this extreme 4% of the time." It does not mean there's a 96% chance the effect is real. In business, p-values appear in A/B tests, market research, and any experiment with a statistical test. The model matters because acting on misinterpreted p-values means treating weak evidence as strong, shipping features that don't work, and building strategy on statistical noise. Understanding what a p-value actually claims — and doesn't — is foundational to honest analytical culture.

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Section 2

How to See It

Product
You're seeing it when a product team ships a feature because the A/B test hit p < 0.05 — without checking effect size. The p-value said the result was unlikely under the null; it didn't say the effect was meaningful.
Strategy
You're seeing it when a market research report labels a consumer preference "statistically significant" based on a p-value alone. With large enough samples, even trivially small differences become significant.
Operations
You're seeing it when a quality-control test rejects a batch based on a marginal p-value near the 0.05 threshold. The binary pass/fail obscures that the evidence against the batch was barely distinguishable from noise.
Section 3

How to Use It

Never interpret a p-value in isolation. Always pair it with effect size (how big is the difference?) and confidence intervals (what's the range?). Set significance thresholds before running experiments, not after. Understand that p < 0.05 is a convention, not a law of nature — and that with enough tests, you'll generate significant results by chance. Treat p-values as one input into decisions, not the decision itself.
Decision filter
"Is this p-value telling me the effect is real and meaningful, or just that I had enough data to detect something trivially small?"
As a founder
Require your data team to report effect size and confidence intervals alongside every p-value. If the effect is statistically significant but practically meaningless, don't ship it. Set your significance thresholds before experiments run, and hold to them — even when the results are close.
Section 5

Founders & Leaders

Marissa MayerFormer CEO, Yahoo; former VP, Google
At Google, Mayer championed data-driven product decisions, running thousands of A/B tests — including the famous experiment testing 41 shades of blue for link color. The approach revealed both the power and the trap of p-value thinking. With Google's traffic volume, virtually any difference becomes statistically significant; the question shifts from "is the p-value below 0.05?" to "is the effect large enough to matter?" Mayer's tenure illustrated that p-values at scale require discipline: distinguishing statistical significance from practical significance, and avoiding the trap of optimizing for measurable micro-effects while ignoring unmeasurable macro-judgment. Founders should internalize this: p-values tell you whether an effect exists, not whether it's worth acting on.
Section 7

Connected Models

Reinforces
Statistical Significance
Statistical significance is the threshold at which a p-value is deemed "low enough" to reject the null hypothesis. P-values are the mechanism; significance is the decision rule applied to them.
Reinforces
P-hacking
P-hacking is the manipulation of analysis to produce low p-values. Understanding what p-values actually measure reveals why hacking them produces false confidence, not real findings.
Tension
Bayes Theorem
Bayes Theorem asks "what's the probability my hypothesis is true given the data?" — the question most people think p-values answer but don't. The tension between frequentist p-values and Bayesian posteriors is fundamental to analytical decision-making.
Section 8

One Key Quote

"A p-value does not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone."
— American Statistical Association (2016)
Section 11

Summary & Further Reading

A p-value measures how surprising your data would be if the null hypothesis were true — not the probability your hypothesis is correct. Always pair it with effect size and confidence intervals, set thresholds before testing, and never treat it as the sole basis for a decision.
01
The Art of Statistics — David Spiegelhalter (2019)
Book
How to learn from data, including the proper interpretation of p-values.
02
Statistics Done Wrong — Alex Reinhart (2015)
Book
The woefully complete guide to common statistical mistakes, centering on p-value misuse.
03
Trustworthy Online Controlled Experiments — Ron Kohavi et al. (2020)
Book
A/B testing at scale, including proper p-value interpretation in product experiments.

Why this matters next

mental modelsBayes Theorem

P-values applied the Bayes Theorem mental model

mental modelsFalse Positives & False Negatives

P-values applied the False Positives & False Negatives mental model

mental modelsScale

P-values applied the Scale mental model

mental modelsQuality

P-values applied the Quality mental model

mental modelsStatistical Significance

P-values applied the Statistical Significance mental model

mental modelsP-hacking

P-values applied the P-hacking mental model

Frequently asked questions

What is P-values?

P-values is a mental model used for better thinking and decision-making.

How do you apply P-values?

To apply P-values, identify situations where this framework is relevant, then use it as a lens to evaluate your options and decisions. The model is most useful when combined with other complementary mental models.

What category does P-values fall under?

P-values falls under the Mathematics & Probability category of mental models. Other models in this category can be found on the Mathematics & Probability hub page.

Why is P-values important?

P-values is important because it provides a structured way to think about problems that would otherwise be approached with intuition alone. Understanding this model helps you avoid common reasoning errors and make better decisions.

Continue exploring

BT

Mental model

Bayes Theorem

A mathematical framework for updating beliefs based on new evidence, proportiona

BT

Mental model

Black Swan Theory

Rare, unpredictable events with extreme impact that are retrospectively rational

CO

Mental model

Compounding

Small consistent gains accumulate exponentially over time — the most powerful fo

CC

Mental model

Correlation vs Causation

The critical distinction between two variables that move together and one actual

ER

Mental model

Ergodicity

The distinction between ensemble averages and time averages — what works across

EG

Mental model

Exponential Growth

Growth that accelerates proportionally to its current size, producing deceptivel

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Popular Mental Models

First Principles ThinkingOccam's RazorCircle of CompetenceInversionConfirmation BiasSecond-Order ThinkingDunning-Kruger EffectSurvivorship BiasPareto PrincipleOpportunity Cost