A randomized controlled experiment (RCT) is the gold standard for learning whether something causes an outcome. You take a population, randomly assign units to treatment (get the intervention) or control (don't), then compare outcomes. Random assignment balances the groups on average — known and unknown factors — so any difference in outcomes can be attributed to the treatment. Observational data can show correlation; RCTs can support causation. The method comes from medicine and agriculture; it is now central in tech (A/B tests), policy (evidence-based programmes), and product development.
RCTs have limits: they're not always feasible (you can't randomise some interventions), they measure average effects (heterogeneity may matter), and they can be expensive or slow. When feasible, they're the best way to answer "did X cause Y?" The strategic discipline is to use RCTs when the question is causal and the intervention can be randomised, and to use other methods when it can't. Don't confuse correlation with causation; when the stakes are high, run an experiment.
Section 2
How to See It
RCT thinking shows up whenever someone assigns units at random to compare treatment vs control. The diagnostic: is there random assignment? Are outcomes compared between groups that were equivalent at the start? Look for A/B tests, clinical trials, policy pilots with random assignment, and product experiments. When someone says "we tested it" or "the data shows it works," ask: was it randomised? If not, the evidence is weaker — selection or confounding could explain the result. The pattern: randomise → intervene → measure → compare.
Business
You're seeing Randomized Controlled Experiment when a company runs an A/B test on a new feature: users are randomly assigned to see the feature or not, and the team compares retention, revenue, or engagement. The random assignment makes the groups comparable; the difference in outcomes is the causal effect of the feature. The same logic applies to pricing tests, email subject lines, and onboarding flows. When the company skips randomisation and compares "users who chose the feature" to others, that's not an RCT — selection bias can dominate.
Technology
You're seeing Randomized Controlled Experiment when a platform runs experiments on recommendation algorithms, ranking, or UI. Traffic is randomly split; one bucket gets the new algorithm, another gets the old. Metrics are compared. The RCT answers "did the new algorithm cause the change in behaviour?" Without randomisation, changes in metrics could be due to seasonality, other product changes, or drift in the user base. Randomisation isolates the effect of the intervention.
Investing
You're seeing Randomized Controlled Experiment when a fund or accelerator randomises which companies get a given programme (e.g. mentorship, capital) and compares outcomes to a control group. That's rare in venture — most capital is not randomly assigned — but policy and some impact programmes use RCTs to measure programme effect. When an investor claims "our portfolio outperforms because of our value-add," the claim is causal; without randomisation, the outperformance could be selection (they picked better companies).
Markets
You're seeing Randomized Controlled Experiment when a government or NGO runs a pilot with random assignment: some villages get the new policy or programme, others don't. Outcomes are compared. The RCT measures the causal effect of the programme. Observational comparison (e.g. villages that adopted vs didn't) would be confounded by why some adopted and others didn't. Randomisation breaks that confounding. Evidence-based policy increasingly relies on RCTs for exactly this reason.
Section 3
How to Use It
Decision filter
"When the question is 'does X cause Y?' and you can randomise who gets X, run an RCT. When you can't randomise, use other methods but be clear about limits: correlation is not causation. Don't claim causal knowledge from observational data without a credible identification strategy."
As a founder
Use experiments for product, pricing, and growth decisions when you can randomise. A/B test features, onboarding, and campaigns. Random assignment is what makes the test interpretable — you're comparing like to like. The mistake is "testing" without randomisation (e.g. rolling out to one region and comparing to another — regions differ). The second mistake is ignoring sample size and duration: underpowered or too-short experiments give noisy or wrong answers. When you can't run an RCT (e.g. a single strategic bet), be explicit that you're not measuring causation — you're making a bet and will learn from the outcome.
As an investor
Portfolio companies that run rigorous experiments (A/B tests, RCTs) are better at learning what works. When evaluating a company, ask how they test: do they randomise? Do they understand statistical power and confounds? Claims like "we changed X and Y went up" are only causal if the change was randomised or otherwise identified. When you can't run an RCT (e.g. "did our advice cause the outcome?"), avoid causal language. Describe selection and correlation; don't claim causation without an experiment or a strong quasi-experimental design.
As a decision-maker
When someone presents evidence that "X caused Y," ask how they know. Was there random assignment? If not, what confounds could explain the association? Encourage experiments when the decision is repeatable and the intervention can be randomised. When RCTs aren't feasible, use the best available evidence but label uncertainty. Don't treat correlation as causation in high-stakes decisions.
Common misapplication: Claiming causation from non-randomised comparisons. "We did X and Y improved" is often just correlation. Seasonality, other changes, or selection can explain it. Reserve causal language for RCTs or strong quasi-experiments.
Second misapplication: Dismissing RCTs as too slow or too hard. Many decisions can be tested with small, fast experiments. The cost of not experimenting is repeated wrong bets. When you can randomise, do it; when you can't, say so and use other evidence with appropriate caution.
Bezos framed experimentation as core to Amazon's culture: the company runs thousands of experiments — A/B tests on pricing, layout, recommendations, and delivery options. The experiments are RCTs at scale: random assignment, comparison of outcomes, iteration. The discipline is to turn product and business questions into testable hypotheses and to use randomisation so the results are interpretable.
Netflix uses rigorous A/B testing for recommendations, artwork, and product features. Random assignment is standard; the company has built infrastructure and culture around running experiments and interpreting them. Hastings has emphasised data-driven decisions; the RCT is the tool that makes "data-driven" causal rather than correlational. When Netflix tests a new feature, it's comparing like to like so the effect can be attributed to the feature.
Section 6
Visual Explanation
Randomized Controlled Experiment: Randomly assign units to treatment or control. Compare outcomes. The difference is the causal effect of the treatment.
Section 7
Connected Models
The RCT sits at the centre of causal inference and evidence-based decision-making. The connections below either describe the same method in context (A/B testing, scientific method), the problems it solves (correlation vs causation, confounding), or the statistics that support it (Bayes, significance).
Reinforces
Scientific Method
The scientific method is hypothesis, test, revise. The RCT is the strongest form of test when the hypothesis is causal ("X causes Y") and you can manipulate X. Randomisation is the design that makes the test interpretable. The RCT is the scientific method applied to cause and effect with maximum rigour.
Reinforces
Correlation vs Causation
Correlation is association; causation is "X changed Y." Observational data gives correlation. The RCT is designed to give causation: random assignment isolates the effect of X. Correlation vs causation is the problem; the RCT is the solution when feasible. When you can't run an RCT, the distinction reminds you to be cautious about causal claims.
Reinforces
Confounding Factor
A confound is a third variable that affects both treatment and outcome, creating a spurious association. Random assignment breaks confounding: it balances the groups on confounds (known and unknown). The RCT is the design that removes confounding by design. When you can't randomise, you must worry about confounds; the RCT avoids that worry.
Leads-to
A/B Testing
A/B testing is the business and product name for an RCT: users are randomly assigned to variant A or B; outcomes are compared. The same logic: randomisation, comparison, causal interpretation. A/B testing is the RCT applied to features, copy, and campaigns. Rigorous A/B tests are RCTs.
Section 8
One Key Quote
"Randomization is the only means of avoiding bias in the comparison of treatments."
— Ronald Fisher, The Design of Experiments (1935)
Fisher's point: without randomisation, the comparison is vulnerable to bias — selection, confounding, or systematic difference between groups. Randomisation is what makes the comparison fair. The practitioner's job: when the question is causal and you can randomise, do it. When you can't, be explicit that your comparison may be biased and avoid strong causal claims.
Section 9
Analyst's Take
Faster Than Normal — Editorial View
Turn causal questions into experiments. When the decision depends on "does X cause Y?" and you can randomise X, run an RCT. Product, pricing, growth, and process decisions often can be tested. The cost of not testing is repeating the same mistakes. Build the habit: hypothesis → design (random assignment) → run → interpret.
Don't claim causation without randomisation. "We did X and Y went up" is usually correlation. Maybe X caused Y; maybe something else did. Without random assignment (or a strong quasi-experiment), say "we observed that when we did X, Y went up" — not "X caused Y." Reserve causal language for experiments.
Respect power and duration. Underpowered experiments (too few units, too short) give noisy or wrong answers. Plan sample size and duration so you can detect the effect you care about. Stopping early because the result "looks good" can be misleading. Run the experiment as designed, then decide.
Use the best available method when RCTs aren't feasible. Some decisions can't be randomised. For those, use quasi-experiments, natural experiments, or observational data — but label the limits. "This suggests X might matter" is different from "X causes Y." Honesty about inference beats overclaiming.
Section 10
Test Yourself
Is this mental model at work here?
Scenario 1
A company rolls out a new feature to users in Europe and compares their retention to users in North America. They conclude the feature caused higher retention.
Scenario 2
A product team randomly assigns 10% of new users to see a new onboarding flow and 90% to the old flow. After two weeks they compare activation rates between the two groups.
Scenario 3
A founder says 'we changed the pricing and revenue went up 20%, so the new pricing caused the increase.'
Section 11
Summary & Further Reading
Summary: A randomized controlled experiment assigns units at random to treatment or control and compares outcomes to estimate the causal effect of the treatment. Use it when the question is causal and the intervention can be randomised; when you can't, avoid claiming causation from correlation. Design for adequate power and duration, and interpret results with effect size and uncertainty. Connected ideas include scientific method, correlation vs causation, confounding, A/B testing, and statistical significance.
The foundational text on randomisation and experimental design. Fisher established the logic of random assignment for causal inference in agriculture and beyond.
Practical guide to designing and running RCTs in development and policy. Covers design, implementation, and interpretation. Applicable to business and product when the intervention can be randomised.
Comprehensive treatment of A/B testing and online experiments. Covers design, analysis, and common pitfalls. The tech industry's reference for RCTs at scale.
Pearl on causation in science and data. Explains why correlation isn't causation and how experiments (and causal diagrams) support causal inference. Conceptual foundation for when and why RCTs work.
Reinforces
Bayes Theorem
Bayes theorem updates belief with evidence. The RCT provides the evidence: the likelihood of the observed data under "treatment works" vs "treatment doesn't." Bayesian analysis of RCTs is common; the RCT is the evidence-generating design, and Bayes is one way to interpret the evidence. They work together in evidence-based decision-making.
Tension
Statistical Significance
Statistical significance tells you whether the observed difference could be due to chance. It doesn't tell you the effect is large or important. The RCT gives an estimate of the causal effect; significance is one way to assess precision. The tension: chasing significance can lead to p-hacking; ignoring it can lead to acting on noise. Use the RCT for the design; use significance (and effect size) for interpretation.
