Mercury's Blog

There has been quite a furor over the recent results from prediction markets on the US presidential elections. Prediction markets have been mocked, derided, and supporters have backpedalled in their praise. This brings up a question; how can we assess the success of a prediction market?

Fundamentals of Prediction Markets

A prediction market is a futures market. In order for a market to be successful, the first thing it needs are traders… people willing to buy and sell with each other. (Automatic “bots” or algorithms can also be set up to trade with human traders.) There need to be a sufficient number of people trading in order to get a good market; I found in my
research that the minimum is around 15-16 people, though this can and will vary depending on the market’s structure.

Another aspect of a markets success is liquidity, generally viewed through the crude measurement of bid/ask spreads. (Liquidity really should take into account both the bid/ask spread and the depth of the order book on either side of a contract, and other factors.) This should come through a sufficient population of traders for a given contract, but that may not always be so.

Finally, traders in a market should not be homogeneous in their sources of information and their analysis. It doesn’t matter if a market on a Democratic political contract is being traded only by Republicans… a market doesn’t necessarily need diversity in the demographic sense. But it does need diversity in traders’ sources of information and/or their analysis of that information. If everyone in a market is using the same data sources and the same analytical tools and processes, it will certainly affect the quality of the forecast.

Different types of Prediction Markets

We have to recognise that there are two basic and different types of prediction markets. There are winner-take-all (binary) markets, and there are index (linear) markets. Winner-take-all markets have two or more options, and when settled, one option is cashed out at $100 and all the rest at $0. Index markets consist of only one contract, and are cashed out according to the actual value of a particular metric at a particular point in time.

Example winner-take-all market: Which party’s presidential nominee will with the 2008 Presidential election?

Example index market: How many Electoral College votes will the winning party’s nominee win in the 2008 Presidential election?

The public rarely has problems interpreting the results of an index market, as error can be measured for each individual market. For example, the Hollywood Stock Exchange traders predict a film will gross $25 million in its first weekend. If it grosses $28 million, the traders were off by $3 million. If other analysts predicted a first weekend gross of $35 million, it would be clear that the market was more accurate. (Alternately, an analyst could have predicted $27 million and be more accurate.) Either way, it is easy to “keep score” on index market predictions.

Where the difficulties lie are in understanding probabilistic forecasts.

Problem #1 – Understanding Probabilities

It can be incredibly difficult to understand probabilities. While we can sense them intuitively (such as your willingness to bet on the flip of a coin), when told a percentage it can be difficult to visualise and understand. One of the hardest things to do is understand how often you can lose when betting on a favourite. Let’s do some simple math:

Chance of winning – Chance of losing
98% – 1 in 50 favourites will lose
95% – 1 in 20 favourites will lose
90% – 1 in 10 favourites will lose
80% – 1 in 5 favourites will lose
75% – 1 in 4 favourites will lose
67% – 1 in 3 favourites will lose

What this also means is that prediction markets have to be “wrong” in order to be right. If all of the contracts trading at 80% actually occurred, the market would be incorrect; 1 in five contracts trading at 80% has to lose!

When thinking back to the last election and prediction markets on Senate seats, many of the “favorites” were trading in the 50-60% range. For the market to be correct, a little less than 1 in 2 of these would need to lose!

To quote Panos Ipeirotis,

If the frontrunner in a prediction market was always the winner, then the markets would have been a seriously flawed mechanism.

[...]

In fact, I would like to argue that the late streak of successes of the markets to always pick the winner of the elections lately has been an anomaly, indicating the favorite bias that exists in these markets. The markets were more accurate than they should, according to the trading prices.

As Panos points out later in the same post, if that bias was consistent in the markets, it would be very lucrative for a trader to simply purchase contracts for the favourites!

Problem #2 – Prediction timescale

Another perception issue that some have with prediction markets is that of timescale. While we call them prediction markets because they provide predictions, those predictions change based on new information! The further away we are from an event, the more uncertain we will be of the outcome. On an even more practical note, the further ahead of time we trade, the more money we have tied up in that contract. (And many exchanges don’t pay you interest in the money you have tied up with them.)

So when do we judge how accurate a prediction market is? Do we take the price from the week before an event? The day before? For an election, do we take it when the polls open? When the polls close?

In my opinion, it all comes down to your goals. InTrade lets traders trade contracts until a winner is settled, because they want an active and accurate marketplace. This has allowed contracts to swing wildly through the day, perhaps most notably in the 2004 Presidential election when leaked exit polls in the afternoon indicated a strong showing for Kerry, only to see actual results not match up with these polls. Other markets look to generate forecasts, so they would end at the point where the information from the forecast was required.

Is this a major problem? I don’t think so. But we do have to recognise that people may be looking at different timescales to suit their needs.

Problem #3 – Assessing accuracy

One of the BIGGEST issues I have with people is regarding accuracy of probabilistic forecasts. Simply put, each contract/market is just a data point, and not a success or failure on its own. This is completely different than an index market, which can be assessed individually.

A binary contract is either correct or it isn’t; there’s no good way to assess the quality of a single data point. What we do is assess the calibration of the marketplace. Of all the contracts judged with a 20% probability, do they happen 20% of the time? Of all the contracts judged with a 95% probability, do they happen 95% of the time? With sufficient data we can draw a calibration curve to determine accuracy in this manner. Prediction markets typically do quite well here.

David Pennock makes a good point in his post on the topic of prediction market assessment, saying:

for a predictor to be considered good it must pass the calibration test, but at the same time some very poor or useless predictors may also pass the calibration test.

[...]

good predictions are not just well calibrated: good predictions are, in some sense, both variable AND well calibrated.

Let’s assume that Barack Obama and Hillary Clinton are the only two candidates running for the Democratic nomination. I could forecast a 50% probability of each of them winning and be perfectly calibrated. But the information would be useless. In liquid and active prediction markets, this typically isn’t a problem. But it certainly is still a factor to consider.

Problem #4 – Compared to what?

Finally, I believe part of the problem with prediction markets is what they’re being compared to. In many peoples’ minds, they’re being compared to an all-knowing crystal ball. For these people, the favourite must always win, or the market was wrong. (Despite the math shown above.)

Prediction markets should be compared to other forecasting methods, and not perfection. Let’s match up prediction markets against the cable-news talking heads and see who’s better. (I haven’t done so, but I would suggest that prediction markets would perform well.)

As Lance Fortnow says,

Notice that when we have a surprise victory in a primary, like Clinton in New Hampshire, much of the talk revolves on why the pundits, polls and prediction markets all “failed.” Meanwhile in sports when we see a surprise victory, like the New York Giants over Dallas and then again in Green Bay, the focus is on what the Giants did right and the Cowboys and Packers did wrong. Sports fans understand probabilities much better than political junkies – upsets happen occasionally, just as they should.

When looking back at the most recently “failure” of the prediction markets (Obama in New Hampshire), I try to think who predicted that Hillary would win? Many top polls had double-digit leads for Obama, and he had “momentum” after his Iowa win. Sure, in hindsight everyone found the reason they thought Obama lost and Hillary won, but where were they the day before and the day of the New Hampshire primary?

Prediction markets aren’t a magic wand or crystal ball. Again, they’re a forecasting method, and should be compared to other forecasting methods and not perfection. There may very well be a better mechanism for forecasting events than a prediction market; I just don’t believe I’ve found one yet.

(There is one paper out there that finds if a trader “adjusts” polls for certain biases, it makes polls far more competitive with prediction markets. It’s a very interesting, though early, result. More comments from Bob Erikson and Justin Wolfers can be found on this website here.

Summary – How have the political prediction markets really performed?

We can’t tell right now. My intuition is that they’re not doing too badly. But as I mentioned many times above, we really need more data from many more contests/contracts in this election. Only then will we be able to tell how accurately they performed.

Until then, we have to accept them for what they are: real-time forecasts of the future, affected by information (including polls, news reports and analysis), and subject to traders’ “irrational exuberance.” In the end, they’re the best dashboard of the current state of an election we have.

This entry was posted on Friday, January 25th, 2008 at 1:42 pm and is filed under General, Real-money Markets.