Psychology & Behavior
Section 1
Core Idea
Automation bias is the tendency to over-trust automated systems — algorithms, dashboards, tools — and to under-check or override our own judgment when the system suggests an answer. We substitute "the system said so" for "is this right?" The core idea: automation is a decision aid, not a replacement for scrutiny. Know when to trust and when to verify.
Free playbook
Get The Business Model Playbook
58 business models, one visual page each: how the money flows, the metrics that matter, and who runs it. Free when you join the Faster Than Normal email.
Free. No spam. Unsubscribe anytime.
Section 2
How to See It
Operations & Ops
You're seeing Automation Bias when teams follow a tool's recommendation (routing, prioritisation, scoring) without questioning edge cases or when the output is clearly wrong. "The system said so" ends the discussion.
Product & Data
You're seeing Automation Bias when a metric or model drives a launch or kill decision and no one asks whether the metric is measuring the right thing or the model is valid for this case.
Section 3
How to Use It
Treat automation as input, not verdict. Define when human review is required (e.g. high stakes, edge cases, low confidence). Build in sanity checks and overrides. When the system disagrees with strong evidence, investigate; don't assume the system is right because it's automated.
Decision filter
"Are we accepting this because the tool said it, or because we've validated it? For high-stakes or unusual cases, did we apply human judgment?"
As a founder
Use automation to scale judgment, not to replace it. Set thresholds for when humans must review. Reward people who flag wrong or weird outputs instead of blindly following the system.
Section 5
Founders & Leaders
Carl IcahnInvestor; activist
Icahn is known for digging into numbers and governance rather than trusting screens or consensus. Founders can apply this by using tools for speed and scale but insisting on human verification for material decisions — and by rewarding those who catch automation errors.
Section 7
Connected Models
Reinforces
Authority Bias
We defer to authority; automation is a kind of system authority. Both bias us toward "they/it said so." Counter both by asking for the reasoning and evidence behind the output.
Tension
Algorithm Aversion
Some people under-trust algorithms. Automation bias is over-trust. The tension: aim for calibrated trust — use the tool where it's validated, verify where it's not or where stakes are high.
Leads-to
Goodhart's Law
When a measure becomes a target, it ceases to be a good measure. Automation bias can make the measured outcome the only thing that gets attention. Use automation without letting the metric fully define the goal.
Section 8
One Key Quote
— James Reason, on human factors"Automation can create a new class of errors — when the human fails to intervene in the face of an automated error because they have learned to trust the system."
Section 11
Summary & Further Reading
Automation bias is over-trusting automated systems and under-using human judgment. Counter it by treating automation as input, defining when humans must review, and verifying high-stakes or odd results. Use tools to scale judgment, not to replace it.