Skip to content
Faster Than Normal
Computer Science & Algorithms

Optimization Algorithms

Model #0669Category: Computer Science & AlgorithmsDepth to apply:
4 min read

Contents

  1. 1. Core Idea
  2. 2. How to See It
  3. 3. How to Use It
  4. 4. Founders & Leaders
  5. 5. Connected Models
  6. 6. One Key Quote
  7. 7. Summary & Further Reading
Computer Science & Algorithms
Section 1

Core Idea

Optimization algorithms search for the best solution in a space of possibilities: maximise reward or minimise cost. Greedy methods (e.g. hill climbing) move toward better neighbours and stop at a local optimum — fast but can get stuck. Random restart runs hill climbing from many starts to improve the chance of finding a global optimum. Simulated annealing allows occasional downhill moves early on, then "cools" to reduce randomness and settle near a good solution. Metropolis and related MCMC methods use probabilistic acceptance to explore the space. The tradeoff: exploitation (refining the current best) vs exploration (trying new regions). Different algorithms tune that tradeoff differently.

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Section 2

How to See It

Deciding & Judging
You're seeing Optimization Algorithms when a process improves a metric step by step (hill climbing), or when a team deliberately tries multiple starting points or allows temporary setbacks to escape local optima (restart, annealing).
Understanding & Analyzing
You're seeing Optimization Algorithms when ML training or parameter tuning uses gradient descent, random restarts, or temperature schedules — the same ideas as hill climbing and simulated annealing in abstract space.
Section 3

How to Use It

When the landscape has many local optima, pure hill climbing can trap you. Use random restart (many independent runs) or simulated annealing (allow bad moves early, then reduce randomness). When the objective is noisy or expensive to evaluate, balance exploitation and exploration explicitly. Match the algorithm to the problem: smooth single peak → hill climbing; rugged landscape → restart or annealing.
Decision filter
"Are we stuck at a solution that's good locally but might not be global? If yes, we're in local-optimum territory — try restarts or annealing (explore more early, then refine). If the space is simple, greedy may be enough."
As a founder
Strategy and product search are optimization: you're climbing a landscape of outcomes. If you're stuck, you may be at a local max. Try restarts (new segments, new positioning) or annealing (experiment with "worse" short-term moves to test new regions). Avoid over-exploiting one hill before exploring the terrain.
Section 5

Founders & Leaders

Jim SimonsFounder, Renaissance Technologies; mathematician
Simons's edge came from optimization at scale: search over strategies, parameters, and data with rigorous use of statistics and algorithms. Renaissance uses the same conceptual toolkit — local search, exploration vs exploitation, and handling noise — in quantitative markets. Founders can adopt the mindset: treat strategy as an optimization over a rugged landscape; use restarts and structured exploration to avoid getting stuck at local optima.
Section 7

Connected Models

Reinforces
Global & Local Maxima
Optimization algorithms are the machinery for finding maxima. Hill climbing finds local maxima; random restart and simulated annealing improve the chance of reaching a global one. The model is the landscape; the algorithms are how we search it.
Tension
Explore-exploit Tradeoff
Hill climbing is pure exploit; annealing and restarts add exploration. The tension: too much exploit → stuck locally; too much explore → never converge. Algorithms formalise how to balance the two.
Leads-to
Theory of Constraints
TOC finds the bottleneck and improves there — a form of local optimization. Optimization algorithms generalise: when improvement at the bottleneck stalls, consider it a local optimum and "restart" or "anneal" (e.g. relax a different constraint) to explore the system.
Section 8

One Key Quote

"Simulated annealing is a general approach to combinatorial optimization that allows occasional acceptance of worse solutions in order to escape local optima." The cooling schedule tunes exploration over time.
— Kirkpatrick et al., Simulated Annealing (1983)
Section 11

Summary & Further Reading

Optimization algorithms search for best solutions: hill climbing (greedy), random restart (many runs), simulated annealing (controlled exploration then refinement). Use them when the landscape has local optima; match the algorithm to the ruggedness of the problem.
01
Introduction to Algorithms — CLRS (2009)
Book
Standard reference for greedy algorithms, local search, and complexity of optimization.
02
Optimization by Simulated Annealing — Kirkpatrick et al. (1983)
Paper
Foundational paper on simulated annealing and escaping local optima.
03
Algorithms to Live By — Christian & Griffiths (2016)
Book
Applies optimization algorithms (including stopping and exploration) to life and business decisions.

Why this matters next

mental modelsExplore-exploit Tradeoff

Optimization Algorithms applied the Explore-exploit Tradeoff mental model

mental modelsTheory of Constraints

Optimization Algorithms applied the Theory of Constraints mental model

mental modelsScale

Optimization Algorithms applied the Scale mental model

mental modelsOptimization Algorithms

Optimization Algorithms applied the Optimization Algorithms mental model

mental modelsAlgorithms

Optimization Algorithms applied the Algorithms mental model

mental modelsCost

Optimization Algorithms applied the Cost mental model

Frequently asked questions

What is Optimization Algorithms?

Optimization Algorithms is a mental model used for better thinking and decision-making.

How do you apply Optimization Algorithms?

To apply Optimization Algorithms, identify situations where this framework is relevant, then use it as a lens to evaluate your options and decisions. The model is most useful when combined with other complementary mental models.

What category does Optimization Algorithms fall under?

Optimization Algorithms falls under the Computer Science & Algorithms category of mental models. Other models in this category can be found on the Computer Science & Algorithms hub page.

Why is Optimization Algorithms important?

Optimization Algorithms is important because it provides a structured way to think about problems that would otherwise be approached with intuition alone. Understanding this model helps you avoid common reasoning errors and make better decisions.

Continue exploring

AB

Mental model

Abstraction

The practice of hiding complexity behind a simpler interface, enabling reasoning

ET

Mental model

Explore-exploit Tradeoff

The fundamental tension between gathering new information (exploring) and levera

ML

Mental model

Metcalfe's Law

The value of a network grows proportionally to the square of the number of its u

ML

Mental model

Moore's Law

The observation that transistor counts on integrated circuits double roughly eve

MM

Mental model

Mythical Man Month

Brooks's Law: adding people to a late software project makes it later due to com

TD

Mental model

Technical Debt

The accumulated cost of expedient decisions in software and systems that must ev

Get Faster Than Normal by email

Ideas from founders and companies.

Free newsletter. Unsubscribe anytime.

Or open the full subscribe page.

Popular Mental Models

First Principles ThinkingOccam's RazorCircle of CompetenceInversionConfirmation BiasSecond-Order ThinkingDunning-Kruger EffectSurvivorship BiasPareto PrincipleOpportunity Cost