Strategy. Innovation. Brand.

IARPA

My Good Judgment

Who's the better forecaster?

Who’s the better forecaster?

We’re all familiar with the idea of placing a bet on a football match. You can bet on many different outcomes: which team will win; by how much; how many total goals will be scored on so on. With a large number of bettors, the aggregate prediction is often remarkably accurate. It’s what James Surowiecki calls The Wisdom of Crowds.

Prediction markets aim to do the same thing but broaden the scope. Instead of betting on sports, they bet on political or economic or natural events. For instance, What’s the probability that: Greece will exit the Euro in 2015; or that nuclear weapons will be used in the India/Pakistan conflict before 2018; or that Miami will have more than 100 flood days by 2020?

The forecasting questions are quite precise and always bounded by a time limit. There should be no question whether the event happens or not. In other words, we can actually judge how accurate the forecasts are.

Why is judging so important? As Philip Tetlock pointed out in Expert Political Judgment: How Good Is It? How Can We Know?, we traditionally don’t measure the accuracy of expert political predictions. Pundits make predictions and nobody checks them. Indeed, Tetlock argues that most pundits make predictions as a way to advertise their consulting businesses. The bolder the prediction, the more powerful the ad.

When Tetlock actually measured the accuracy of expert political predictions, he discovered they were essentially useless. Tetlock writes, “The results were startling. The average expert did only slightly better than random guessing.” Remember that the next time you read an expert prediction.

You may remember that I wrote about a prediction market – InTrade – during the 2012 elections in the United States. Based in Ireland, InTrade allowed people all over the world to place bets on who would win the presidential election, as well as various Senate, gubernatorial, and congressional elections. InTrade’s electoral predictions were remarkably accurate. (It did less well in predicting Supreme Court decisions).

Unfortunately, the U.S. government saw InTrade as a form of online gambling. As such, it needed to be tightly regulated or perhaps even suppressed. It’s a complicated story — and may have involved “financial irregularities” on InTrade’s part — but, in 2013, InTrade decided to close its doors.

So, how can we use prediction markets in the United States? In its wisdom, another agency of the federal government, the Intelligence Advanced Reesarch Projects Activity (IARPA), started a prediction tournament called Aggregative Contingent Estimation (ACE). IARPA/ACE has run a prediction tournament for the past three years. Various teams – mainly from academic institutions – participate for the honor of being named the most accurate forecaster.

And who wins these tournaments? A team called The Good Judgment Project (GJP) put together by none other than Philip Tetlock. GJP selects several thousand volunteers, gives them some training on how to make forecasts, and asks them to forecast the several hundred questions included in the IARPA/ACE tournament.

The Good Judgment project wins the tournament consistently. They must be doing something right. And who is the newest forecaster on the Good Judgment team? Well… with all due modesty, it’s me.

To say the least, I’m excited to participate – and I expect to write about my experiences over the coming months. I can predict with 70% confidence that I won’t be a world class forecaster in the first go round. But I may just learn a thing or two and improve my accuracy over time. Wish me luck.

My Social Media

YouTube Twitter Facebook LinkedIn

Newsletter Signup
Archives