Statistical Significance

You're seeing it when a product manager ships a feature because the A/B test reached significance, without checking whether the 0.3% conversion lift is worth the engineering maintenance cost. Significant doesn't mean meaningful.

You're seeing it when a pitch deck claims "statistically significant user growth" from a 200-person pilot. With a small sample and many possible metrics, significance may have been achieved by chance.

You're seeing it when a campaign is declared successful because one of eight measured KPIs reached p < 0.05. With eight tests, you'd expect roughly one false positive by chance alone.

Build an experimentation culture that demands practical significance alongside statistical significance. A result that's "significant" with a 0.2% lift on millions of users may not justify the cost. Teach your team to ask "significant and so what?" — not just "significant, ship it."

Lütke built Shopify's product culture around experimentation — but with an emphasis on practical significance over statistical significance. Shopify runs thousands of experiments across its merchant platform, and the team learned early that statistically significant results at Shopify's scale were trivially easy to generate. A 0.1% change in checkout behavior is statistically significant with millions of transactions — but not worth the engineering and cognitive cost of maintaining the change. Lütke pushed teams to define minimum detectable effects that were practically meaningful before running tests, ensuring that significance served the business rather than just the analytics dashboard. Founders should set the same standard: decide what magnitude of effect would change your behavior before you look at the data.