Wisdom of Crowd

In 1906, Francis Galton attended a livestock fair in Plymouth where 787 people paid sixpence each to guess the weight of an ox. Galton — an elitist who believed expertise should determine policy — expected the crowd's guesses to be wildly inaccurate. He was wrong. The median guess was 1,207 pounds. The actual weight was 1,198 pounds. The crowd was off by less than 1%. No individual expert came closer. The statistician who trusted expertise over democracy had accidentally discovered the most powerful argument against expertise as the sole basis for judgement.

James Surowiecki formalised the insight in The Wisdom of Crowds (2004), identifying four conditions under which collective judgement outperforms individual experts: diversity of opinion (each person holds private information, even if it is only an eccentric interpretation of known facts), independence (each person's opinion is not determined by the opinions of others), decentralization (people draw on local knowledge and specialised expertise), and aggregation (a mechanism exists for turning individual judgements into a collective answer). When all four conditions hold, the errors in individual judgements cancel out and the remaining signal converges on truth with startling precision. When any condition fails, the crowd becomes a mob — and mobs are reliably worse than individuals.

Prediction markets are the cleanest modern demonstration. The Iowa Electronic Markets have outperformed major polls in predicting US presidential elections in 15 of 17 cycles since 1988. Prediction markets work because they satisfy all four conditions: diverse participants (traders with different information, different analytical frameworks, different biases), independence (each trader places their own bet without coordinating), decentralization (traders operate from different locations with different information sources), and aggregation (the market price is the mechanism that synthesises individual judgements into a single number). The price is not an opinion. It is the distillation of thousands of opinions, each weighted by the confidence the holder places in it — because in a prediction market, you back your belief with money.

Wikipedia demonstrates the same principle in knowledge production. The encyclopedia that anyone can edit has been shown in multiple studies to rival Encyclopaedia Britannica in accuracy across scientific topics. The mechanism is not that Wikipedia's individual editors are smarter than Britannica's experts. The mechanism is that the sheer diversity and volume of editors — each catching different errors, adding different knowledge, correcting different biases — produces a continuously self-correcting document that converges on accuracy through aggregation. The process looks chaotic from the outside. The output is reliable because the conditions are met: diverse editors, largely independent judgement, decentralised expertise, and a platform that aggregates contributions into a coherent article.

The critical insight — and the one most people miss — is that the conditions are demanding. Most "crowds" are not wise. They are herds. When diversity breaks down (everyone reads the same sources, follows the same influencers, shares the same priors), the crowd loses the error-cancelling diversity that makes it intelligent. When independence breaks down (people observe and copy each other's judgements rather than forming their own), the crowd converges on the first prominent signal rather than on truth. When aggregation fails (there is no mechanism to synthesise individual opinions into a collective judgement), the diversity exists but cannot be harvested. Financial bubbles, social media pile-ons, and political polarisation are all examples of crowds that fail the conditions — and crowds that fail the conditions are not just unintelligent. They are systematically worse than the individuals who compose them, because the herd amplifies errors rather than cancelling them.