AboutHow we built thisSponsorshipShop
SearchSubscribeDecision ToolsBusiness ModelsFrameworksReading Lists
Privacy PolicyTerms of UseCookie PolicyRefund PolicyAccessibilityDisclaimer

© 2026 Faster Than Normal. All rights reserved.

Faster Than Normal
DecisionsPeopleBusinessesNewsletterSubscribe
Start reading →
  1. Home
  2. Mental models
  3. Reliability of Case Evidence
Mathematics & Probability

Reliability of Case Evidence

Model #0788Category: Mathematics & ProbabilityDepth to apply:
5 min read

On this page

  • Core Idea
  • How to See It
  • How to Use It
  • Founders & Leaders
  • Connected Models
  • One Key Quote
  • Summary & Further Reading

Contents

  1. 1. Core Idea
  2. 2. How to See It
  3. 3. How to Use It
  4. 4. Founders & Leaders
  5. 5. Connected Models
  6. 6. One Key Quote
  7. 7. Summary & Further Reading
·Mathematics & Probability
Section 1

Core Idea

Reliability of case evidence asks how much weight a single case — one customer story, one founder's success path, one market data point — should carry in forming beliefs. Individual cases are vivid, memorable, and persuasive, but they carry enormous hidden variance. A single successful customer doesn't prove product-market fit. A single founder's strategy doesn't generalize to all companies. The model warns that the human brain overweights narrative case evidence relative to its actual informational value, especially when the case is dramatic or emotionally resonant. Reliable conclusions require multiple independent observations, not compelling anecdotes. The question isn't whether the case is true — it's whether it's representative.
Section 2

How to See It

Strategy
You're seeing it when a board member advocates entering a new market because "Company X did it and tripled revenue." One case — no matter how successful — doesn't establish that the strategy works in general or for you specifically.
Sales
You're seeing it when a sales team builds an entire persona around one enthusiastic early customer. That customer's needs may be idiosyncratic, not representative of the broader market.
Hiring
You're seeing it when a hiring decision is justified by "our best engineer came from a non-traditional background, so let's change all our hiring criteria." One success story doesn't validate a systematic policy change.
Section 3

How to Use It

When case evidence is presented, ask three questions: Is this case representative or exceptional? How many similar cases produced different outcomes? What's the base rate for this type of success? Use cases for hypothesis generation — they suggest what might work — but require larger samples for hypothesis validation. Train your team to distinguish between "this happened" and "this is likely to happen again."
Decision filter
"Is this a representative case or a compelling outlier, and do I have enough cases to draw a reliable conclusion?"
As a founder
Use customer case studies and competitor examples to generate hypotheses, not to confirm them. Before changing strategy based on a single compelling case, ask how many counterexamples exist that you haven't encountered. Seek the base rate before trusting the anecdote.
Section 5

Founders & Leaders

Daniel EkFounder & CEO, Spotify
Ek built Spotify's recommendation and content strategy on aggregate data patterns rather than individual case evidence. Early music-industry wisdom was driven by case-based reasoning: "This artist broke out this way, so that's the formula." Ek's insight was that individual breakout stories were unreliable guides to systematic strategy — survivorship bias made every success story look replicable when most attempts following the same path failed. Spotify instead built systems that analyzed millions of listening patterns to identify reliable signals, treating each individual data point as one observation among many rather than a standalone conclusion. Founders benefit from the same discipline: let individual cases inform intuition, but let aggregate evidence drive decisions.
Section 7

Connected Models

Reinforces
Survivorship Bias
Survivorship bias is what makes case evidence unreliable — you see the cases that succeeded and never encounter the larger population that failed using the same approach.
Reinforces
Law of Small Numbers
The law of small numbers describes the tendency to draw confident conclusions from tiny samples. Reliability of case evidence warns against this specifically when the "sample" is a single vivid case.
Tension
Anecdotal Fallacy
The anecdotal fallacy dismisses case evidence entirely. Reliability of case evidence is more nuanced — it doesn't reject cases, but asks how much weight they deserve relative to systematic evidence.
Section 8

One Key Quote

"The confidence people have in their beliefs is not a measure of the quality of evidence but of the coherence of the story the mind has managed to construct."
— Daniel Kahneman
Section 11

Summary & Further Reading

Case evidence is vivid and persuasive but carries hidden variance. A single case generates hypotheses; it doesn't validate them. Ask whether the case is representative, seek the base rate, and demand multiple independent observations before changing strategy on one compelling story.
01
Thinking, Fast and Slow — Daniel Kahneman (2011)
Book
How cognitive biases lead us to overweight vivid case evidence and underweight statistical base rates.
02
The Black Swan — Nassim Nicholas Taleb (2007)
Book
Why individual dramatic cases mislead us about the underlying distribution of outcomes.
03
Spotify Untold — Sven Carlsson & Jonas Leijonhufvud (2021)
Book
How Spotify built data-driven strategy that moved beyond case-based music-industry wisdom.

Why this matters next

mental modelsSurvivorship Bias

Reliability of Case Evidence applied the Survivorship Bias mental model

mental modelsBayes Theorem

Reliability of Case Evidence applied the Bayes Theorem mental model

mental modelsNarrative

Reliability of Case Evidence applied the Narrative mental model

mental modelsLaw of Small Numbers

Reliability of Case Evidence applied the Law of Small Numbers mental model

mental modelsIntuition

Reliability of Case Evidence applied the Intuition mental model

mental modelsQuality

Reliability of Case Evidence applied the Quality mental model

Frequently asked questions

What is Reliability of Case Evidence?+

Reliability of Case Evidence is a mental model used for better thinking and decision-making.

How do you apply Reliability of Case Evidence?+

To apply Reliability of Case Evidence, identify situations where this framework is relevant, then use it as a lens to evaluate your options and decisions. The model is most useful when combined with other complementary mental models.

What category does Reliability of Case Evidence fall under?+

Reliability of Case Evidence falls under the Mathematics & Probability category of mental models. Other models in this category can be found on the Mathematics & Probability hub page.

Why is Reliability of Case Evidence important?+

Reliability of Case Evidence is important because it provides a structured way to think about problems that would otherwise be approached with intuition alone. Understanding this model helps you avoid common reasoning errors and make better decisions.

Continue exploring

BT

Mental model

Bayes Theorem

A mathematical framework for updating beliefs based on new evidence, proportiona

BT

Mental model

Black Swan Theory

Rare, unpredictable events with extreme impact that are retrospectively rational

CO

Mental model

Compounding

Small consistent gains accumulate exponentially over time — the most powerful fo

CC

Mental model

Correlation vs Causation

The critical distinction between two variables that move together and one actual

ER

Mental model

Ergodicity

The distinction between ensemble averages and time averages — what works across

EG

Mental model

Exponential Growth

Growth that accelerates proportionally to its current size, producing deceptivel

More like this, in your inbox

I send a newsletter every week — free, no spam, unsubscribe anytime.

Or open the full subscribe page.

On this page

  • Core Idea
  • How to See It
  • How to Use It
  • Founders & Leaders
  • Connected Models
  • One Key Quote
  • Summary & Further Reading

Popular Mental Models

First Principles ThinkingOccam's RazorCircle of CompetenceInversionConfirmation BiasSecond-Order ThinkingDunning-Kruger EffectSurvivorship BiasPareto PrincipleOpportunity Cost