·Mathematics & Probability
Section 1
The Core Idea
Extreme outcomes tend to be followed by less extreme ones. This single statistical fact is responsible for more misallocated resources, more false confidence, and more broken strategies than any other concept in probability.
Regression to the mean describes the phenomenon whereby a variable that is extreme on its first measurement will tend to be closer to the average on its second measurement — not because of any causal force pulling it back, but because extreme values are, by definition, partly the product of luck, and luck does not persist. The tall parent's child is still likely tall, but less extremely so. The fund manager who crushed the market last year will probably beat it again, but by a smaller margin. The team that won fourteen consecutive games will probably keep winning, but not fourteen more in a row. The mathematics are not optional. They are a consequence of any system where outcomes are influenced by a combination of skill and randomness.
Francis Galton discovered the phenomenon in the 1880s while studying hereditary height. He measured the heights of 928 adult children and their parents and found that children of unusually tall parents were, on average, shorter than their parents — and children of unusually short parents were taller. Galton initially called it "regression toward mediocrity," a phrase that inadvertently captured both the statistical truth and the psychological resistance it provokes. The finding unsettled him. It seemed to imply that humanity was converging toward a grey uniformity. It was not. The distribution of heights stayed the same across generations — the variance was preserved. What changed was which individuals occupied the extremes. The extreme position was not heritable because it was partly the product of chance factors that did not repeat.
The mathematics are straightforward. Any measured outcome can be decomposed into a stable component (skill, genetics, structural advantage) and a variable component (luck, noise, random error). When the variable component contributes to an extreme measurement, it is overwhelmingly likely that the variable component was favourable — because unfavourable randomness combined with the stable component would have produced a less extreme result. On the next measurement, the variable component is drawn independently. It might be favourable again, but the expected value of any random variable is its mean, not its previous realisation. So the expected second measurement is closer to the overall average than the first. The regression is not toward the individual's true ability. It is toward the population mean, because the noise that inflated the first measurement has nowhere to go but back.
The concept sounds elementary, and it is — mathematically. The difficulty is entirely psychological. Humans are narrative creatures. When a CEO delivers an extraordinary quarter, the board constructs a causal story: the new strategy worked, the team executed brilliantly, the product resonated. When the next quarter is merely good, another causal story emerges: the market shifted, execution slipped, focus was lost. Both stories may contain elements of truth. But the most parsimonious explanation — that an extreme outcome was followed by a less extreme one because that is what extreme outcomes do — never makes it into the boardroom. Regression to the mean has no protagonist, no villain, no narrative arc. It is statistically inevitable and psychologically invisible.
Daniel Kahneman identified regression to the mean as one of the most consequential cognitive blind spots in decision-making. In Thinking, Fast and Slow, he recounted a pivotal moment from his work with Israeli Air Force flight instructors. The instructors told Kahneman that praise after an exceptionally good landing was typically followed by a worse landing, while criticism after a poor landing was followed by an improvement. They concluded — reasonably, by their experience — that criticism was more effective than praise. Kahneman recognised the error instantly: the pattern was pure regression to the mean. An unusually good landing would be followed by a more typical one regardless of what the instructor said, and an unusually bad landing would likewise be followed by improvement. The instructors' feedback was irrelevant. But the temporal coincidence between punishment and improvement created a powerful and entirely false causal belief that punishment works and praise backfires. The instructors were not stupid. They were victims of a statistical phenomenon that mimics causation so convincingly that it fools experts in every domain.
The phenomenon operates everywhere outcomes combine ability with randomness — which is to say, everywhere. Sports performance, academic test scores, investment returns, business metrics, medical treatments, crime rates, employee evaluations, student grades, weather patterns, and political approval ratings all exhibit regression to the mean. The phenomenon does not require any change in underlying quality. It does not imply that the system is deteriorating or improving. It is a mathematical artefact of measurement in the presence of noise, and it will appear in any dataset where the correlation between successive measurements is less than perfect.
The consequences for decision-making are severe. Companies fire managers after a bad year that was partially driven by noise and hire replacements who benefit from regression and get credit for the improvement. Sports teams trade players after a slump and the replacement performs better — not because of superior talent, but because the original player's slump was an extreme measurement that would have regressed anyway. Governments implement policies after a crisis — a spike in crime, a disease outbreak, a market crash — and claim credit when conditions improve, when the improvement was baked in by the same statistical logic that guaranteed the extreme would not persist.
Consider the scale of the distortion in medicine alone. Before the widespread adoption of randomised controlled trials, most medical treatments were evaluated by administering them to patients who were at their worst — because that is when patients seek treatment — and observing whether they improved. They almost always improved. Not because the treatments worked, but because extreme symptoms regress toward the patient's baseline regardless of intervention. Bloodletting persisted for two thousand years partly because physicians observed that patients who were bled tended to get better afterwards. They did get better. They would have gotten better without the bleeding. The regression was invisible because there was no control group — no untreated patients whose identical improvement would have revealed the treatment as inert. The randomised controlled trial, developed by Ronald Fisher and applied to medicine by Austin Bradford Hill in the 1948 streptomycin trial, was specifically designed to neutralise regression to the mean by ensuring that both treated and untreated groups regressed equally, so that any remaining difference could be attributed to the treatment. The entire architecture of modern evidence-based medicine is, at its core, a system for distinguishing genuine treatment effects from regression artifacts.
The financial markets provide an equally consequential theatre. The mutual fund industry's most reliable pattern is not alpha generation — it is regression to the mean in fund performance. S&P Global's SPIVA scorecards consistently show that funds in the top quartile over one period are no more likely to remain in the top quartile over the next period than chance would predict. The "hot hand" in fund management is almost entirely a regression illusion: the fund that outperformed was partly skilled and partly lucky, and the lucky component does not persist. Yet the entire fund marketing apparatus — the five-star ratings, the performance advertisements, the "top fund" lists — is built on presenting extreme recent performance as if it were a reliable predictor. Investors allocate billions of dollars annually to funds at the peak of their noise component, and then experience the inevitable regression as a personal loss.
The Israeli flight instructor story is not an anecdote. It is a template for how regression to the mean systematically distorts cause-and-effect reasoning in every institution that evaluates performance, allocates resources, or intervenes after extreme outcomes. The cost is not merely intellectual. It is operational: organisations repeatedly punish people for bad luck, reward people for good luck, and build their entire incentive systems around a statistical illusion.
The depth of the illusion is hard to overstate. Kahneman himself called regression to the mean "the most important lesson I have ever learned" — a remarkable statement from the psychologist who also identified loss aversion, anchoring, and the availability heuristic. He ranked regression above all of them because of its unique combination of ubiquity and invisibility: it operates in every domain, affects every decision that follows an extreme outcome, and is almost never recognised in real time by the people it is affecting.
