Mercury's Blog

There is a huge mis-conception in the media when it comes to evaluating the success of prediction markets in the recent US election. Simply put, you have to be wrong to be right. But depending on the type of prediction market you are operating, different methods of assessment are required. I’m going to go through each type of market and compare it to the best-of-class of poll aggregators (fivethirtyeight.com), discuss what they came up with, and potentially find a potential winner from the 2008 forecasts.

Probabilistic predictions (and prediction markets)

The most popular prediction markets this election season were probabilistic markets, where the payoff was either 1 or 0. (In the case of InTrade, $10 or $0.) The market price can and is interpreted as a percent probability that the contract will take place; one of the candidates getting elected. Unfortunately, many commentators believe that once a contract is above 50% that the candidate will win, and if a candidate is above 50% in a given market and doesn’t win, the prediction market has failed.

This is completely wrong.

I’ve discussed in previous posts that you have to be wrong in order to be right. As a quick reminder:

• 98% chance = 1 in 50 will be wrong
• 90% chance = 1 in 10 will be wrong
• 80% chance = 1 in 5 will be wrong
• 75% chance = 1 in 4 will be wrong
• 67% chance = 1 in 3 will be wrong
• 50% chance = 1 in 2 will be wrong

So how did the prediction markets do? And how did fivethirtyeight.com do in comparison? (They’re the only site I know of that takes poll results and turns them into probabilistic forecasts mathematically.) Here’s a quick snapshot:

InTrade results

We immediately run into the primary problem with evaluating these markets… while there are 50+ markets, only a small number of these were not at the extremes of the scales. I’ve just included those with a market price of <90%.

This looks great for InTrade, but it's deceiving. For example, let's take a look at all contracts at approximately 80%. (FL, OH, VA, NV, and CO). They have an average market price of 82.6%. If InTrade was perfectly calibrated, one of these should have been wrong.

It looks better on the other side. If you calculate MT, GA, ND, and IN they have an average market price of 26.75%. In fact, one of these four were wrong (Indiana).

For the mathematical amongst you, it should be clear that this type of assessment is quite crude. When you have so few data points, you need to pick and choose "bins" of data with your own best judgement. Ideally there would be enough markets that you could create strict "bins" of data and measure against those. (This is exactly how Inkling created their plots here… I would encourage you to read their post, too.) Unfortunately, we only have a small number of “battleground” markets and this just isn’t possible.

Fivethirtyeight.com results

In my opinion, fivethirtyeight.com appears to have done marginally better. Take the bottom five markets shown, which have a combined probability of 15%. In fact, one of these five actually occurred; a 20% success rate.

Another slice of data shows the same thing: MT, ND, IN, and MO have an average probability of 30% and an actual success rate of 25%. Taking MO, NC and FL the probability was 61%, and and actual success rate of 67%.

I say that fivethirtyeight.com has done marginally better because I could take reasonable slices of data throughout their predictions and come up with reasonably calibrated results. I had to specifically pick and choose to find similar results with InTrade. In other words, the fivethirtyeight.com forecasts were more internally consistent.

Non-probability predictions (and prediction markets)

Some of the lesser-cited and more lightly-traded prediction markets for the 2008 election cycle were on index markets on the vote share for each candidate. Here was the final tally of the national vote:

• Democrat – Obama – 52.7%
• Republican – McCain – 46.0%
• Other – 1.3%

What did the Iowa Electronic Market forecast? (Data from midnight on the 3rd)

• Democrat – Obama – 53.5% (0.8% error)
• Republican – McCain – 46.4% (0.4% error)

What did fivethirtyeight.com forecast?

• Democrat – Obama – 52.3% (0.4% error)
• Republican – McCain – 46.2% (0.2% error)
• Other – 1.5% (0.2% error)

To be fair, fivethirtyeight.com was the best of the “poll aggregators”. Real Clear Politics and Pollster.com came up with the following:

• Democrat – Obama – 52.1%, 52.0%
• Republican – McCain – 44.5%, 44.4%

It’s clear from this data that while the Iowa Electronic Markets were quite accurate, fivethirtyeight.com forecasts were even more accurate.

For reference, redbluerichpoor.com showed the following result from fivethirtyeight.com’s final forecasts (of state vote share) which were pretty accurate:

The issue of time

Recently, George Neumann of the Iowa Electronic Markets sent out a document to the Prediction Markets Google group that claimed that the IEM “continue to dominate polls.” One specific line really struck me as ridiculous:

During this 886 day period the average absolute error was 1.2%, amazingly similar to the final polling results but for a much longer period.

So now we’re supposed to assess the accuracy of a prediction market over a 2+ year period?!? So as the race moves and swings, and the markets with it, the IEM seems only to be concerned about the average. This implies that the election is much more static than I believe it to be.

Face it, the race changes. If the election was held within a week of the Republican convention, John McCain would have likely won… the markets reflected that. But events change, and the market changes along with them. This is why I disagree with Peter McClusky’s analysis of price changes here.

Uncertainty is priced into prediction market prices; in the fall Nate Silver of fivethirtyeight.com was consistently predicting a much higher probability of Obama winning the race than prediction markets, because the uncertainty of events between the forecast date and the election was priced into the market. As the election date got closer the uncertainty was removed from the price and Obama’s price went up. But this happens rather late in election futures; in election after election we’ve seen examples of late-breaking news that has had the ability to shift the outcome of a race.

Where does this leave prediction markets?

Were prediction markets consistently better than a well-performing site like fivethirtyeight.com? No.

Were prediction markets consistently worse than a site like fivethirtyeight.com? No.

They perform largely the same, though the final accuracy of fivethirtyeight.com was a tad bit better than InTrade/IEM. But the purpose of the two types of sites are different.

Fivethirtyeight.com and similar sites take current data and process it to extrapolate trends. These sites lag real events; Nate Silver mentioned on a number of times that he expected his model’s forecast to move, but that it hadn’t because the relevant polls hadn’t hit the model yet. Between the time a poll closes, the result released and then incorporated into the model is anything from a day or two to several days. So while it looks quite accurate, a poll aggregator is a lagging indicator. (Another couple of election cycles will tell us if their accuracy continues or was just a fluke in 2008.)

Prediction markets show the results of what a group of traders believe what will happen. This includes polling data, but also reacts to real-time information. A candidate makes a huge gaffe, and the market price will reflect it in minutes, where a poll aggregator could take days to see any effect.

This is the “social utility” of prediction markets, to answer a question that Chris Masse always poses. While they are on par with the accuracy of the best poll aggregators, their forecasts are real-time and reflect the state of the race right now. No other mechanism does this. While markets are certainly fed by polls, that isn’t the whole puzzle in and of itself.

Prediction markets worked quite well again this election cycle. Though their final forecasts were on par with the best poll aggregators, their real-time forecasts throughout the election season is the reason why they should be examined and discussed more broadly.

Tomorrow is going to be a landmark day for prediction markets. The 2008 US election cycle has been the most-polled, most-predicted, and likely the most-analyzed election in history. It’s been going for nearly two years, and I for one am glad the election will soon be over and governing (by whomever wins) will soon begin.

But why will it be a landmark day for prediction markets? Simply put: the data.

Prediction Markets and Polls – The Data

There are prediction markets on a wide variety of sites, with both play-money and real-money incentives. Iowa Electronic Markets, InTrade, and Betfair for real-money; HubDub, Inkling, NewsFutures for play-money. There are more, but these are the sites I’ve seen cited most often. (It’s too bad ConsensusPoint didn’t push TheWSX.com this election cycle.)

More importantly, there are a few sites that offer incredibly deep (and also probabilistic) analysis into polls. Most notably fivethirtyeight.com, which I seem to be checking a couple of times a day, now. There are national polls, national tracking polls, state polls, and even some state tracking polls! Fivethirtyeight in particular does deep-level statistical analysis to determine from poll results and demographic data how likely each state is to vote for each candidate.

The number of data points, from different prediction markets, polls, and poll trendlines/analysis will be immense. The sum total of data that will be available after this election should be a treasure trove for researchers, and should finally prove the accuracy of prediction markets.

But there’s a hitch…

Yes, there’s always a hitch, and it’s something I’ve discussed before. In elections, polls and prediction markets are measuring two different things.

Polls are measuring the percentage support for a candidate. Generally around 40-60% or so, unless it’s a total blow-out.

Prediction markets measure the percentage chance that the candidate will win their election. When the election is tight, around 50%, when it’s a blowout can regularly be 95%+. (Few prediction markets exist for candidates’ vote share; really only on the presidential level.)

What should you expect on Election Day?

I expect that a number of news outlets will be quoting percentages from InTrade in the run-up to the end of polls closing in the evening. Already final results contests are springing up, including one in the New York Times where you earn points based on current InTrade odds. You can also expect a LOT of volume on the markets tomorrow. But once the results start rolling in, the news is going to focus on the candidates alone. Wednesday will start the morning-after evaluation of the polls and markets, which will likely last for quite some time.

What does this mean in the end?

Comparing the results of polls and prediction markets is certainly like comparing apples and oranges. There are certainly some similarities, but they are fundamentally different.

What we need to do is evaluate how each forecasting method performed independently. For prediction markets, that means that a “failure” (where a prediction >50% didn’t happen) is quite likely a success. For polls, that means that a result just a few percentage points off (outside its MOE) is a failure.

I believe that prediction markets will come out looking quite good in this election. They’ve already proven their worth to me; when poll results might indicate a close or tightening race in places, the prediction market magnifies the difference, and in many cases demonstrates the poll volatility is just noise.

In the end, are the results from the prediction markets useful? Based on the number of times I’ve seen them cited this year… the answer is an unqualified YES. Are they perfect? No, and neither is any other forecasting system or technique.

I’m really looking forward to tomorrow…

The US election is just over a week away, and with that there are a few different topics I’d like to touch on. With the explosion in new prediction markets since the last presidential election, we should see some interesting (but hopefully consistent) results.

• First, a great post from Koleman Strumpf on Midas Oracle points out that half of all trades on the Betfair exchange in 2004 occurred on Election Day! While I personally think that was quite likely due to the early exit poll news for Kerry and the subsequent swing back to Bush, it proves that there are still quite a few people that may be waiting until the very last day to trade.
• Jason Ruspini just started a new thread on the Prediction Markets e-mail group regarding some divergences he’s seen between prices on InTrade and fivethirtyeight.com.

  While I think some of the things he’s observed is due to the way Nate Silver presents data on his site, Jason brings up a very good point. A thorough analysis of movement in the InTrade prediction markets should be compared to the daily calculated win percentage from fivethirtyeight (where all data comes strictly from polls). I think it could be very revealing, and give the public quite a bit more data on the accuracy of polling, aggregation of polls, and prediction markets.

• A long time ago I started four different markets on Inkling Markets that will hopefully predict control over each house of Congress, and the number of seats each party will have after the election. Data is shown here:
  
  
  (as I write this, the Democratic percentage is 53.8%, which corresponds to 234 seats in the House of Representatives.)
  
  
• I may need to update my post on prediction market software soon. Xpree (founded by Mat Fogarty, and recently joined by Leslie Fine, a well known prediction market researcher from HP) may be changing their name. The top three picks according to their contest on NameThis were:
  1. Metricast
  2. UREprojection
  3. Keymet

  Personally, I don’t like #2 or #3, but Metricast sounds interesting. It also sounds like a much better fit to what they do than “Xpree.”

  Good luck to them, if they choose to go down this route.

• Do you speak Danish? Nosco is hiring!
  For English speakers, so is InTrade and Xpree.

I’ve got a few posts coming soon on the innovation topics I’ve been discussing more recently, but prediction markets have really been the hot topic with the current election. Here are a few issues I find interesting:

Obama on InTrade

The recent nomination of Sarah Palin as Republican VP candidate seemed to have changed the dynamics of the race, and she has certainly sucked up quite a bit of media attention (both positive and negative).

What I find curious is the results on the InTrade prediction markets. Specifically, over the past day and a half there have been some really interesting swings on the Obama (and McCain) contracts.

I can absolutely understand that the candidates are getting closer. But Obama dipping below 50%, and then McCain going above Obama are pretty big steps! I’m surprised they happened without a particularly major event taking place. That said, I have a few reasons/thoughts:

• Manipulation: It’s certainly not unknown, but is certainly hard to detect. (Because it comes down to intention instead of action). But when I read Mike Linksvayer’s post at Midas Oracle noting that the Obama-President and DemNominee-President weren’t consistent, I do tend to think that it might be an irrational or potentially manipulative trader. That said, at least on InTrade the two contracts are now about equivalent for McCain, but about 2.5% different for Obama.
• Trader bias: In the discussions to Mike’s post, Nigel makes a good point that other real-money exchanges (such as Betfair and the Iowa Electronic Markets) still show Obama as a favorite. IEM currently has the Winner-Take-All, aka probability market, at 55/45 Obama. Are the traders on InTrade biased somehow? (Again, this could link back to manipulation depending on their motivations).
• Volatility: I’ll admit, I don’t check InTrade on a daily basis. But that said, it seems that the presidential contracts are quite volatile right now. Considering that no major events have been happening recently, I find it a bit odd that we’re seeing five-point swings in the matter of hours. Perhaps there is an extra sensitivity around the 50% price-point?

Minnesota and the IEM

Iowa is right next door to Minnesota (the state where I grew up), and Minnesota has a hotly contested election for a seat in the US Senate up for grabs this year. The Republican incumbent is Norm Coleman, and the Democratic challenger is comedian/radio-show-host Al Franken. Iowa Electronic Markets set up a market for this race, which I haven’t seen marketed particularly widely. Then again, it’s largely local/regional interest.

I find the difference between the IEM prices and InTrade to again be interesting in this market. In the local and US-legal-without-question market on the IEM, Coleman leads Franken 62/38. On InTrade, Coleman leads Franken at 57/43. There looks to be a more volume dollar-wise on InTrade, but IEM potentially has more local knowledge here.

Should be interesting to follow this in the next couple of months and see how and when the markets converge.

exchangeP

If you’re reading this blog, I figure there’s a decent chance you would have heard about TechCrunch and the TechCrunch50 conference. Fifty startups launched and demo’d their new sites and technologies over three days.

I got a kick looking at exchangeP. (Check out their site with demo video here.) Essentially they have created a niche prediction market site to determine the valuation of private companies, such as Facebook, LinkedIn, etc. While they describe it as a fantasy stock market, it’s really a fantasy prediction market since each “stock” is eventually cashed out.

Part of me is a bit incredulous that a site like this managed to get accepted at a conference like TechCrunch. Turning it into a profitable business seems to be something that they weren’t really considering.

But what really impressed and amused me was Mark Cuban’s feedback to them. (You can see it on the video.) The correct and impressive insight is that he remarked that it would never be big enough to turn it into a real-money site, so it wasn’t really viable as a business that way. Amusingly, he recommended that since they’ve already built their market tool/technology that they could then become a provider of technology for all of the people/companies that want to start prediction markets of their own.

So perhaps I’ll be adding them to my list of prediction market software vendors soon? Then again, they probably don’t know the diversity of prediction market software vendors out there today. From my first look at their site, it looks sub-standard.

I’m fascinated when real-money markets can be directly compared to play-money markets, particularly when I can potentially make some money.

For those of you that haven’t read my research paper, I created a series of play-money prediction markets on rowing events in the summer of 2006. The results were as what you might expect; quite accurate. When there were just 16 or more traders involved in a market, the results could be relied upon for good forecasts. I’ve turned that project into a longer-running project, which has also taken place last summer and this summer. (I’m collecting data to analyze how the number of traders required has changed since Inkling adopted Robin Hanson’s MSR, which wasn’t in place for the original research.) It’s proven quite popular amongst amateur rowers in the UK.

[I'd like to thank Inkling for providing the platform; I plan to analyze and publish the results of the last two summers' research this fall once the Olympics markets are closed out.]

A few weeks ago I created a prediction market for each of the 14 Olympic rowing events. A few days ago I was checking Betfair and realised that they had also created markets for all 14 Olympic rowing events. (Both to determine the winners as well as the podium places for each event.) When I compared the two marketplaces there were some potentially profitable discrepancies.

There seem to be two types of market-makers in the Betfair markets. One simply entered lay bets for each entrant at very poor odds (ie, 1.01 decimal) just so there was something to trade. That’s not very useful. But another type of market-maker entered more realistic odds, but odds that were dramatically skewed toward long-shots. For example, in the Men’s Double Sculls event, the play-money prediction market forecasted a win for New Zealand with about an 80% probability. However, I was able to buy shares on Betfair at 2.56, or about 39% probability! While the over-round on the market was quite large, it was because the odds on the long-shots were unreasonable. Odds on the favourites in these markets were very good, and this was the case on as many as half of the events.

So if my play-money markets are accurate (as I expect them to be), I should be able to make a little money on the Olympics, courtesy of the initial market-makers on Betfair. Some people may argue that the play-money predictions won’t be as accurate because they don’t involve real money, but looking at the current Betfair market there’s so little liquidity to have quality forecasts. Unfortunately the lack of book depth means that the mis-pricing won’t last long and won’t be hugely profitable in absolute terms, but that should be interesting to watch.

I think this is an excellent example of where a play-money market can profitably inform real-money market trading.

Questions, comments? Please enter your comments below. (No registration or sign-up required.)

(Reminder: You can subscribe to receive new posts here by e-mail, just click here.)