Mathematics & Probability
Section 1
Core Idea
Statistical significance is the threshold at which an observed result is deemed unlikely enough under the null hypothesis to be considered "real" rather than random noise. Conventionally set at p < 0.05, it means there's less than a 5% chance of seeing this data if the null hypothesis were true. The concept is a decision rule, not a truth detector. It doesn't measure the size of an effect, its practical importance, or the probability that the hypothesis is correct. Yet in business, "statistically significant" is routinely treated as a stamp of certainty — a green light to ship features, change pricing, or scale campaigns. The model teaches that significance is the floor of evidence, not the ceiling, and that the threshold itself is a convention with no special scientific authority.
Free playbook
Get The Business Model Playbook
58 business models, one visual page each: how the money flows, the metrics that matter, and who runs it. Free when you join the Faster Than Normal email.
Free. No spam. Unsubscribe anytime.
Section 2
How to See It
Product
You're seeing it when a product manager ships a feature because the A/B test reached significance, without checking whether the 0.3% conversion lift is worth the engineering maintenance cost. Significant doesn't mean meaningful.
Fundraising
You're seeing it when a pitch deck claims "statistically significant user growth" from a 200-person pilot. With a small sample and many possible metrics, significance may have been achieved by chance.
Marketing
You're seeing it when a campaign is declared successful because one of eight measured KPIs reached p < 0.05. With eight tests, you'd expect roughly one false positive by chance alone.
Section 3
How to Use It
Set your significance threshold before running experiments, and hold to it. Always pair significance with effect size — a significant but trivially small effect isn't worth acting on. Correct for multiple comparisons when testing more than one hypothesis. Treat significance as one input, not a binary go/no-go signal. When the result is significant but the effect is small, or the sample is enormous, apply practical judgment before shipping.
Decision filter
"Is this result both statistically significant and practically meaningful enough to warrant action?"
As a founder
Build an experimentation culture that demands practical significance alongside statistical significance. A result that's "significant" with a 0.2% lift on millions of users may not justify the cost. Teach your team to ask "significant and so what?" — not just "significant, ship it."
Section 5
Founders & Leaders
Tobi LütkeFounder & CEO, Shopify
Lütke built Shopify's product culture around experimentation — but with an emphasis on practical significance over statistical significance. Shopify runs thousands of experiments across its merchant platform, and the team learned early that statistically significant results at Shopify's scale were trivially easy to generate. A 0.1% change in checkout behavior is statistically significant with millions of transactions — but not worth the engineering and cognitive cost of maintaining the change. Lütke pushed teams to define minimum detectable effects that were practically meaningful before running tests, ensuring that significance served the business rather than just the analytics dashboard. Founders should set the same standard: decide what magnitude of effect would change your behavior before you look at the data.
Section 7
Connected Models
Reinforces
P-values
P-values are the mechanism that determines statistical significance — the result is "significant" when the p-value falls below the chosen threshold. Understanding p-values is prerequisite to interpreting significance correctly.
Reinforces
A/B Testing
A/B testing is the most common business application of statistical significance. The test framework generates the data; significance is the decision rule applied to it.
Tension
Confidence Intervals
Confidence intervals provide richer information than binary significance — they show the range of plausible effects, not just whether the effect crossed a threshold. They're the antidote to significance's binary thinking.
Section 8
One Key Quote
— Jacob Cohen"The earth is round (p < 0.05)."
Section 11
Summary & Further Reading
Statistical significance is a decision rule for distinguishing signal from noise, not a measure of truth or practical importance. Always pair it with effect size, correct for multiple comparisons, and define what "meaningful" looks like before you look at the data.
01
Book
Definitive guide to running and interpreting A/B tests, including statistical significance.
02
Book
Comprehensive guide to the misuse of statistical significance in research and business.
03
Book
Clear explanation of statistical concepts for general audiences, including the meaning of significance.