Skip to content
Faster Than Normal
Mathematics & Probability

Statistical Significance

Model #0790Category: Mathematics & ProbabilityDepth to apply:

By Updated 3 sources

4 min read

Contents

  1. 1. Core Idea
  2. 2. How to See It
  3. 3. How to Use It
  4. 4. Founders & Leaders
  5. 5. Connected Models
  6. 6. One Key Quote
  7. 7. Summary & Further Reading
Mathematics & Probability
Section 1

Core Idea

Statistical significance is the threshold at which an observed result is deemed unlikely enough under the null hypothesis to be considered "real" rather than random noise. Conventionally set at p < 0.05, it means there's less than a 5% chance of seeing this data if the null hypothesis were true. The concept is a decision rule, not a truth detector. It doesn't measure the size of an effect, its practical importance, or the probability that the hypothesis is correct. Yet in business, "statistically significant" is routinely treated as a stamp of certainty — a green light to ship features, change pricing, or scale campaigns. The model teaches that significance is the floor of evidence, not the ceiling, and that the threshold itself is a convention with no special scientific authority.

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Section 2

How to See It

Product
You're seeing it when a product manager ships a feature because the A/B test reached significance, without checking whether the 0.3% conversion lift is worth the engineering maintenance cost. Significant doesn't mean meaningful.
Fundraising
You're seeing it when a pitch deck claims "statistically significant user growth" from a 200-person pilot. With a small sample and many possible metrics, significance may have been achieved by chance.
Marketing
You're seeing it when a campaign is declared successful because one of eight measured KPIs reached p < 0.05. With eight tests, you'd expect roughly one false positive by chance alone.
Section 3

How to Use It

Set your significance threshold before running experiments, and hold to it. Always pair significance with effect size — a significant but trivially small effect isn't worth acting on. Correct for multiple comparisons when testing more than one hypothesis. Treat significance as one input, not a binary go/no-go signal. When the result is significant but the effect is small, or the sample is enormous, apply practical judgment before shipping.
Decision filter
"Is this result both statistically significant and practically meaningful enough to warrant action?"
As a founder
Build an experimentation culture that demands practical significance alongside statistical significance. A result that's "significant" with a 0.2% lift on millions of users may not justify the cost. Teach your team to ask "significant and so what?" — not just "significant, ship it."
Section 5

Founders & Leaders

Tobi LütkeFounder & CEO, Shopify
Lütke built Shopify's product culture around experimentation — but with an emphasis on practical significance over statistical significance. Shopify runs thousands of experiments across its merchant platform, and the team learned early that statistically significant results at Shopify's scale were trivially easy to generate. A 0.1% change in checkout behavior is statistically significant with millions of transactions — but not worth the engineering and cognitive cost of maintaining the change. Lütke pushed teams to define minimum detectable effects that were practically meaningful before running tests, ensuring that significance served the business rather than just the analytics dashboard. Founders should set the same standard: decide what magnitude of effect would change your behavior before you look at the data.
Section 7

Connected Models

Reinforces
P-values
P-values are the mechanism that determines statistical significance — the result is "significant" when the p-value falls below the chosen threshold. Understanding p-values is prerequisite to interpreting significance correctly.
Reinforces
A/B Testing
A/B testing is the most common business application of statistical significance. The test framework generates the data; significance is the decision rule applied to it.
Tension
Confidence Intervals
Confidence intervals provide richer information than binary significance — they show the range of plausible effects, not just whether the effect crossed a threshold. They're the antidote to significance's binary thinking.
Section 8

One Key Quote

"The earth is round (p < 0.05)."
— Jacob Cohen
Section 11

Summary & Further Reading

Statistical significance is a decision rule for distinguishing signal from noise, not a measure of truth or practical importance. Always pair it with effect size, correct for multiple comparisons, and define what "meaningful" looks like before you look at the data.
01
Trustworthy Online Controlled Experiments — Ron Kohavi et al. (2020)
Book
Definitive guide to running and interpreting A/B tests, including statistical significance.
02
Statistics Done Wrong — Alex Reinhart (2015)
Book
Comprehensive guide to the misuse of statistical significance in research and business.
03
The Art of Statistics — David Spiegelhalter (2019)
Book
Clear explanation of statistical concepts for general audiences, including the meaning of significance.

Why this matters next

mental modelsScale

Statistical Significance applied the Scale mental model

mental modelsStatistical Significance

Statistical Significance applied the Statistical Significance mental model

mental modelsCost

Statistical Significance applied the Cost mental model

mental modelsTransaction

Statistical Significance applied the Transaction mental model

mental modelsKPI

Statistical Significance applied the KPI mental model

mental modelsA/B Testing

Statistical Significance applied the A/B Testing mental model

Frequently asked questions

What is Statistical Significance?

Statistical Significance is a mental model used for better thinking and decision-making.

How do you apply Statistical Significance?

To apply Statistical Significance, identify situations where this framework is relevant, then use it as a lens to evaluate your options and decisions. The model is most useful when combined with other complementary mental models.

What category does Statistical Significance fall under?

Statistical Significance falls under the Mathematics & Probability category of mental models. Other models in this category can be found on the Mathematics & Probability hub page.

Why is Statistical Significance important?

Statistical Significance is important because it provides a structured way to think about problems that would otherwise be approached with intuition alone. Understanding this model helps you avoid common reasoning errors and make better decisions.

Continue exploring

BT

Mental model

Bayes Theorem

A mathematical framework for updating beliefs based on new evidence, proportiona

BT

Mental model

Black Swan Theory

Rare, unpredictable events with extreme impact that are retrospectively rational

CO

Mental model

Compounding

Small consistent gains accumulate exponentially over time — the most powerful fo

CC

Mental model

Correlation vs Causation

The critical distinction between two variables that move together and one actual

ER

Mental model

Ergodicity

The distinction between ensemble averages and time averages — what works across

EG

Mental model

Exponential Growth

Growth that accelerates proportionally to its current size, producing deceptivel

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Popular Mental Models

First Principles ThinkingOccam's RazorCircle of CompetenceInversionConfirmation BiasSecond-Order ThinkingDunning-Kruger EffectSurvivorship BiasPareto PrincipleOpportunity Cost