Field Testing

Focus groups loved New Coke. Two hundred thousand blind taste tests confirmed that consumers preferred the new formula to both old Coke and Pepsi. Coca-Cola launched with total confidence in April 1985. Within seventy-nine days, they brought back the original formula and absorbed one of the most expensive product failures in corporate history. The taste tests were not wrong — in a controlled setting, people genuinely preferred the sweeter formula. The taste tests were irrelevant. They measured preference in a vacuum. They could not measure brand attachment, nostalgic identity, the psychological ownership people felt over "their" Coke, or the revolt that erupts when you change something woven into cultural fabric. The lab said yes. The field said no. The field was right. Field testing — validating your product or idea with real users in real conditions, not in a laboratory or conference room — is the discipline of closing the gap between what people say they will do and what they actually do. That gap has killed more products than bad engineering ever will.

Drew Houston understood the gap in 2007. He had built the earliest version of Dropbox but could not convince investors that consumers wanted cloud file storage — the concept felt abstract, the market already had competitors, and the problem seemed solved by USB drives. So Houston made a three-minute demonstration video showing exactly how Dropbox worked: drag a file into a folder, watch it appear on another computer. He did not commission a survey. He did not convene a focus group. He posted the video on Hacker News — a community dense with his target users — and measured whether people would act. Within twenty-four hours, 75,000 people had signed up for the beta waitlist. The video was a field test: a real product concept, shown to real potential users, in the real environment where they made decisions, measured by the only metric that matters — whether they moved. No amount of market research could have generated that signal. The signal required real people making a real commitment — even one as small as entering an email address — in the context where they actually lived.

Nick Swinmurn ran the same logic with shoes. In 1999, he wanted to know whether people would buy footwear online — a proposition that every retail expert and every focus group would have dismissed. People need to try on shoes. They need to feel the material. They need to check the fit. Swinmurn ignored the objections and ran the test. He walked into shoe stores around San Francisco, photographed the inventory, posted the photos on a basic website, and waited. When someone placed an order, he went back to the store, bought the shoes at full retail price, and shipped them himself. Zero inventory. Zero warehouse. Zero risk beyond his time. The field test measured one variable: would real people spend real money on shoes they had not tried on? They did. The company Swinmurn built on that signal was Zappos. Amazon acquired it for $1.2 billion in 2009.

The power of field testing is ecological validity — a concept from experimental psychology that describes how well findings from one setting predict behaviour in another. Laboratory experiments maximise internal validity by controlling variables. Field tests sacrifice some control for massive gains in ecological validity — the fidelity of the signal to real-world conditions. For business decisions, this trade-off is almost always worth making. The variables that determine whether a product succeeds — willingness to pay, switching costs, competitive alternatives, contextual friction, the chasm between stated preference and revealed behaviour — are precisely the variables that disappear inside a conference room. A focus group can tell you whether people find your concept interesting. Only a field test can tell you whether they will reach for their wallet.