Journal · Evidence
What forecasting tournaments teach about deciding
The largest study of whether judgement can be improved found that it can, cheaply, and told us exactly which procedures did it.
Can you train someone to make better judgements? Yes, measurably. In the IARPA forecasting tournament, roughly an hour of training in probabilistic reasoning and scenario work improved accuracy by about 10% on Brier scores, sustained across four years.
The Good Judgment Project, led by Mellers and Tetlock, ran forecasters against geopolitical questions over several years with scoring against outcomes. Among the findings: a short training module produced a durable improvement, and dispositional factors including open-minded thinking predicted accuracy.
What was in the hour
This is the part worth copying, because it is procedural rather than inspirational:
- State coherent probabilities, respecting the basic rules rather than using vague verbal hedges.
- Explore and challenge your assumptions explicitly, as a step rather than an attitude.
- Identify the causal drivers: what would actually have to move for this to go either way.
- Build the best and worst cases and a sensible interval, rather than a point estimate.
- Avoid over-correcting when new information arrives, which is its own failure mode.
Nothing in that list is a personality. All of it is a procedure, which is why it transfers.
What it implies for business decisions
Business decisions differ from forecasting questions: they are actions, not predictions, and the outcome depends on what you do next. But every one of them contains forecasts, usually unstated. "The pipeline holds through Q3." "This hire ramps in two quarters." Those are predictions, they carry the decision, and they are almost never written down as probabilities where anyone could check them later.
The uncomfortable implication
If judgement is trainable through procedures, then simulating an admired executive is not the mechanism. Nothing in this literature suggests that borrowing a famous person's voice improves anything. What improved accuracy was a checklist, applied consistently, by ordinary people.
Five Peers editorial note. Sources named in the text.
Five procedures, applied consistently, on your actual decision.
Compose my board