Daily catch-up: dart-throwing chimpanzees and how to predict the future

A new book distils the wisdom of an experiment by American intelligence to improve its forecasts

I wrote about the hazards of prediction on Sunday, with reference to what Paddy Power, the betting company, yesterday called “the most dramatic turnaround in recent political betting history” – namely Jeremy Corbyn’s imminent victory in the Labour leadership election.

I quoted a forthcoming book by Philip Tetlock and Dan Gardner, Superforecasting, but I didn’t have space to do it justice. It is a terrific piece of work that deserves to be widely read.

Not least because it is this month’s winner of my Acknowledgements Prize. In his acknowledgements, Gardner, Tetlock’s Canadian co-author, thanks his editor, his wife, his mother and Queen Elizabeth. “Amanda [Cook] edited this book and without her sometimes astonishing patience and perseverance it would not exist. Ditto my wife, Sandra, because I’m useless without her, and my mother, June, who made me, which was inarguably a necessary antecedent to my participation in this book’s creation. And the Queen is the Queen, dammit. Long may she reign.”

The book describes the results of an extraordinary experiment sponsored by the Intelligence Advanced Research Projects Activity (IARPA), by which the agency sought to improve the ability of US intelligence to predict world events.

Between 2011 and 2015, it required five teams of researchers, including Tetlock’s, to submit daily forecasts in answer to questions set by IARPA. The project collected a million measurable predictions, allowing the teams to be assessed and their methods improved.

In year one, Tetlock boasts that his team beat the control group by 60 per cent. In year two it was 78 per cent and doing so much better than the University of Michigan and MIT that IARPA dropped the other teams.

Thus he had proved that, although the average expert forecaster may perform no better than a dart-throwing chimpanzee, some forecasters can be consistently better than others. The book examines what it is about so-called superforecasters that makes them better.

When the conclusions are distilled to Tetlock’s 10 Commandments (wittily, there are 11 of them, just to make the point that you can never predict that people will do what they say they are going to do), they are disappointingly obvious. Such as: “Strike the right balance between under- and over-confidence.”

Yet this is a simple judgement that anyone making a forecast often overlooks. When I ruled out Corbyn’s chances, I was relying on out-of-date assumptions and giving no weight to indications to the contrary.

But then, according to Tetlock’s eighth Commandment, I have to “beware of rearview-mirror hindsight biases”. He writes that, although the more common error is to learn too little from failure and to overlook flaws in your basic assumptions, it is also possible to learn too much – you may have been basically on the right track but made a minor technical mistake that had big ramifications”.

So it is possible to be better at prediction, but not easy. It cannot be simplified into 10, or even 11, dos and don’ts.

However, the book is wonderful to read, opening with the story of Mohamed Bouazizi, the fruit and veg seller in Tunisia who set himself on fire in December 2010 in protest at police harassment and started the Arab Spring. It discusses chaos theory and Edward Lorenz’s 1972 paper, “Predictability: Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?”

And it has a good line in apposite quotations. Discussing scientific method, Tetlock and Gardner quote Galen, the second-century doctor to Roman emperors: “All who drink of this treatment recover in a short time, except those whom it does not help, who all die. It is obvious, therefore, that it fails only in incurable cases.”

Highly recommended.