P-values

A p-value is the probability of observing data at least as extreme as the actual result, assuming the null hypothesis is true. It does not tell you the probability that your hypothesis is correct. This distinction — universally misunderstood — is the source of enormous decision-making error. A p-value of 0.04 means: "If there were truly no effect, we'd see data this extreme 4% of the time." It does not mean there's a 96% chance the effect is real. In business, p-values appear in A/B tests, market research, and any experiment with a statistical test. The model matters because acting on misinterpreted p-values means treating weak evidence as strong, shipping features that don't work, and building strategy on statistical noise. Understanding what a p-value actually claims — and doesn't — is foundational to honest analytical culture.

Never interpret a p-value in isolation. Always pair it with effect size (how big is the difference?) and confidence intervals (what's the range?). Set significance thresholds before running experiments, not after. Understand that p < 0.05 is a convention, not a law of nature — and that with enough tests, you'll generate significant results by chance. Treat p-values as one input into decisions, not the decision itself.

A p-value measures how surprising your data would be if the null hypothesis were true — not the probability your hypothesis is correct. Always pair it with effect size and confidence intervals, set thresholds before testing, and never treat it as the sole basis for a decision.