Superforecasting

---
name: superforecasting
description: Make probabilistic predictions when strategic decisions require accurate risk assessment
---

# Superforecasting

**Category:** Decision-Making & Strategic Thinking
**Source:** Philip Tetlock & Dan Gardner - "Superforecasting: The Art and Science of Prediction" (2015)
**Practitioner Score:** 46/50 (Tier 1 Canonical)

## Overview

Superforecasting is a systematic methodology for making accurate probabilistic predictions, developed through Philip Tetlock's Good Judgment Project. The research identified "superforecasters" - individuals who consistently outperform experts, pundits, and even intelligence analysts by 30% or more. The framework codifies their techniques into 10 actionable commandments plus deliberate practice protocols.

**Core Insight:** Prediction accuracy is a learnable skill. By combining rigorous process (break problems down, update beliefs incrementally, balance outside/inside views) with calibrated probabilistic thinking and error analysis, anyone can dramatically improve forecasting ability.

**Evidence:** Good Judgment Project participants trained in these techniques beat CIA analysts with classified information access.

## When to Use

- **Strategic decisions** - Market entry, product launches, competitive moves
- **Resource allocation** - Investment decisions, hiring plans, capacity planning
- **Risk assessment** - Project timelines, crisis likelihood, threat analysis
- **Competitive intelligence** - Predicting competitor actions, market shifts
- **Long-range planning** - Technology adoption curves, regulatory changes

**Anti-patterns:**
- Decisions already made (confirmation bias trap)
- Binary yes/no thinking without probability ranges
- One-shot predictions without learning feedback
- High-emotion situations without cooling-off period

## How to Execute

### Step 1: Triage - Choose Forecast-Worthy Questions
**Action:** Focus effort on questions where hard work pays off
- **Skip "clocklike" questions:** Simple rules/trends suffice (e.g., "Will sun rise tomorrow?")
- **Skip "cloud-like" questions:** Too random even for models (e.g., "Will World War III start?")
- **Target Goldilocks zone:** Difficult but tractable with analysis
- **Output:** Prioritized list of forecastable questions

### Step 2: Fermi-Style Decomposition
**Action:** Break intractable problems into tractable sub-problems
- **Identify knowable parts:** What can be researched or estimated?
- **Expose unknowables:** Flush ignorance into the open
- **Example:** "Will product X succeed?" → market size × conversion rate × pricing × competition
- **Output:** Hierarchical breakdown with researchable components

### Step 3: Balance Outside View (Base Rates) and Inside View
**Action:** Start with reference class, adjust for specifics
- **Outside view first:** How often do things of this sort happen in situations of this sort?
- **Find comparison class:** Similar products, markets, technologies
- **Inside view adjustment:** What makes this case unique?
- **Output:** Base rate probability + reasoned adjustments

### Step 4: Incremental Belief Updating
**Action:** Update forecasts as new evidence arrives - not too much, not too little
- **Avoid under-reacting:** Ignoring genuinely new information
- **Avoid over-reacting:** Jumping to conclusions from noisy signals
- **Bayesian mindset:** P(H|E) = P(E|H) × P(H) / P(E)
- **Output:** Revised probability with explicit reasoning

### Step 5: Consider Clashing Causal Forces
**Action:** Map arguments for AND against your thesis
- **Steel-man opposition:** Understand counterarguments deeply
- **Force interaction:** How do conflicting factors balance?
- **Example:** "AI adoption" → Cost savings (pro) vs. Implementation complexity (con)
- **Output:** Two-column list of forces with relative weights

### Step 6: Granular Probability Estimates
**Action:** Translate vague hunches into numeric probabilities
- **Avoid vague language:** "Likely" means what exactly?
- **Use fine gradations:** 55% vs. 60% forces precision
- **Calibration practice:** Track how often your 70% predictions come true
- **Output:** Numeric probability (e.g., 68%) with confidence range

### Step 7: Balance Under/Overconfidence
**Action:** Manage trade-off between decisiveness and humility
- **Calibration:** Are your 80% predictions correct 80% of the time?
- **Resolution:** Can you distinguish 60% from 80% events?
- **Avoid extremes:** "Definitely" (99%+) and "No way" (1%-) rarely justified
- **Output:** Calibrated probability that neither overstates nor understates certainty

### Step 8: Learn from Errors Without Hindsight Bias
**Action:** Analyze mistakes while resisting "I knew it all along"
- **Pre-mortem:** Before outcome, write why forecast might fail
- **Post-mortem:** After outcome, compare to pre-mortem (not current knowledge)
- **Brier score tracking:** Measure accuracy over time
- **Output:** Error log with root cause analysis

### Step 9: Leverage Team Wisdom
**Action:** Master collaborative forecasting dynamics
- **Perspective-taking:** Reproduce others' arguments to their satisfaction
- **Precision questioning:** Help clarify without judgment
- **Constructive confrontation:** Disagree without being disagreeable
- **Output:** Team forecast incorporating diverse viewpoints

### Step 10: Master the Error-Balancing Bicycle
**Action:** Treat commandments as guidelines requiring constant judgment
- **No rigid rules:** Every situation is unique
- **Deliberate practice:** Forecasting is skill built through repetition
- **Feedback loops:** Clear, unambiguous results inform learning
- **Output:** Continuous improvement trajectory

## Real-World Examples

**Good Judgment Project (2011-2015):**
- Superforecasters beat intelligence analysts by 30%
- Ordinary people trained in these methods outperformed experts
- Result: Validated that forecasting is a learnable skill

**Prediction Markets (Metaculus, Good Judgment Open):**
- Calibrated forecasters consistently identify probability ranges
- Aggregated predictions outperform individual experts
- Result: Operational use in policy, business, research

**Tech Industry Product Forecasting:**
- Decompose adoption rates into addressable market × conversion × retention
- Update predictions as beta data arrives
- Result: Better resource allocation, realistic roadmaps

## Integration Points

**Complements:**
- **Brier Score:** Measures superforecasting accuracy quantitatively
- **Fermi Estimation:** Powers Step 2 decomposition
- **Bayes' Theorem:** Mathematical foundation for belief updating
- **Calibration:** Essential skill for Step 7 confidence management
- **Base Rate Analysis:** Core of Step 3 outside view

**Contrasts with:**
- **Expert Intuition:** Systematic process vs. gut feel
- **Punditry:** Probabilistic humility vs. confident pronouncements
- **Binary Thinking:** 65% vs. "yes/no"

## Common Pitfalls

**Pitfall 1: Anchoring on Initial Estimate**
- **Warning sign:** Forecast barely moves despite major news
- **Fix:** Explicit belief updating protocol after each information update

**Pitfall 2: Ignoring Base Rates**
- **Warning sign:** "This time is different" without evidence
- **Fix:** Always start with outside view reference class

**Pitfall 3: Overconfidence in Extremes**
- **Warning sign:** Many forecasts at 5% or 95%
- **Fix:** Force justification for extreme probabilities, track calibration

**Pitfall 4: Confirmation Bias in Research**
- **Warning sign:** Only seeking evidence supporting initial view
- **Fix:** Actively search for disconfirming evidence (Step 5)

**Pitfall 5: No Feedback Loop**
- **Warning sign:** Making predictions but never tracking outcomes
- **Fix:** Maintain prediction log with dates, probabilities, and resolutions

## Validation Checklist

- [ ] Question is in Goldilocks zone (neither trivial nor impossible)
- [ ] Problem decomposed into researchable sub-components
- [ ] Base rate identified from reference class
- [ ] Both supporting and opposing forces mapped
- [ ] Probability is numeric and granular (not vague language)
- [ ] Calibration tracked over time (70% predictions = 70% accuracy)
- [ ] Forecast updated as new information arrives
- [ ] Pre-mortem written before outcome known
- [ ] Team input incorporated through structured dialogue

## Key Metrics

**Brier Score:** Primary accuracy measure (0 = perfect, 2 = worst)
- Formula: (1/N) Σ(forecast - outcome)²
- Target: < 0.20 for well-calibrated forecaster

**Calibration:** Do your X% predictions happen X% of the time?
- Plot predicted probability vs. observed frequency
- Perfect calibration = diagonal line

**Resolution:** Can you distinguish different probability levels?
- Difference in outcomes between 60% and 80% forecasts
- Higher resolution = better discrimination

## Further Reading

- **"Superforecasting"** - Philip Tetlock & Dan Gardner (2015)
- **"Expert Political Judgment"** - Philip Tetlock (2005)
- **Good Judgment Open:** Free forecasting platform with training
- **Metaculus:** Advanced forecasting community
- **"The Signal and the Noise"** - Nate Silver (Bayesian thinking)