Mercury's Blog

How did 2009 turn out?

Early this year I posted my predictions for 2009. In the best spirit of Robin Hanson (getting better predictions by simply tracking how close predictions matched reality), I want to see how I did.

• “Prediction markets in 2009 are going to become even more well-known and wide-spread, but there will be no single event that brings them to the attention of the public. It’s going to be a slow, but steady, growth.” – This was spot on. All of the vendors appear to be doing well, but there was no big “event” that brought everyone attention.
• “All of the prediction market vendors will mature their business offering/proposition.” – Not having been on the receiving end of any of their sales pitches, I can’t say this one is true for sure. But from their blog posts and public statements I would assess this as likely.
• “HubDub will continue to only be the only strongly popular play-money prediction market.” – Put me down as wrong on this one. Nigel and the creators of HubDub have focused their time and effort on FanDuel instead. (Rightly, for revenue reasons!) So while HubDub is still active, it’s not the hive of activity it was for a while.

That said, I’m still a fan of niche, public, popular sites like HSX. They managed to turn play-money prediction markets into a real-money revenue stream by analyzing trader behaviour (which can only be seen by administrators) and selling that business intelligence to the studios. This could be replicated in many industries, such as video games or television.

• “While a couple additional software vendors may appear, I get the feeling that the market for prediction market software is largely saturated.” – This was also spot on.
• “I’m looking forward to see how the CantorExchange develops.” – Not so much a prediction, but a hope that it proved interesting. It’s taken a long time to get up and running apparently, and won’t be widely launched for real-money contracts until 2010. (If I read the website correctly.)

A great development from InTrade

Just yesterday John Delaney of InTrade posted on his blog that InTrade will soon be offering some historical market data to the public for free. (As he notes, this means they’re losing a source of potential revenue, as historical data can be quite valuable.) This is a great development, and should be a solid source of data for people to dig in and get interested in how traders operate in a prediction market. Kudos to John for doing this.

Tomorrow is going to be a landmark day for prediction markets. The 2008 US election cycle has been the most-polled, most-predicted, and likely the most-analyzed election in history. It’s been going for nearly two years, and I for one am glad the election will soon be over and governing (by whomever wins) will soon begin.

But why will it be a landmark day for prediction markets? Simply put: the data.

Prediction Markets and Polls – The Data

There are prediction markets on a wide variety of sites, with both play-money and real-money incentives. Iowa Electronic Markets, InTrade, and Betfair for real-money; HubDub, Inkling, NewsFutures for play-money. There are more, but these are the sites I’ve seen cited most often. (It’s too bad ConsensusPoint didn’t push TheWSX.com this election cycle.)

More importantly, there are a few sites that offer incredibly deep (and also probabilistic) analysis into polls. Most notably fivethirtyeight.com, which I seem to be checking a couple of times a day, now. There are national polls, national tracking polls, state polls, and even some state tracking polls! Fivethirtyeight in particular does deep-level statistical analysis to determine from poll results and demographic data how likely each state is to vote for each candidate.

The number of data points, from different prediction markets, polls, and poll trendlines/analysis will be immense. The sum total of data that will be available after this election should be a treasure trove for researchers, and should finally prove the accuracy of prediction markets.

But there’s a hitch…

Yes, there’s always a hitch, and it’s something I’ve discussed before. In elections, polls and prediction markets are measuring two different things.

Polls are measuring the percentage support for a candidate. Generally around 40-60% or so, unless it’s a total blow-out.

Prediction markets measure the percentage chance that the candidate will win their election. When the election is tight, around 50%, when it’s a blowout can regularly be 95%+. (Few prediction markets exist for candidates’ vote share; really only on the presidential level.)

What should you expect on Election Day?

I expect that a number of news outlets will be quoting percentages from InTrade in the run-up to the end of polls closing in the evening. Already final results contests are springing up, including one in the New York Times where you earn points based on current InTrade odds. You can also expect a LOT of volume on the markets tomorrow. But once the results start rolling in, the news is going to focus on the candidates alone. Wednesday will start the morning-after evaluation of the polls and markets, which will likely last for quite some time.

What does this mean in the end?

Comparing the results of polls and prediction markets is certainly like comparing apples and oranges. There are certainly some similarities, but they are fundamentally different.

What we need to do is evaluate how each forecasting method performed independently. For prediction markets, that means that a “failure” (where a prediction >50% didn’t happen) is quite likely a success. For polls, that means that a result just a few percentage points off (outside its MOE) is a failure.

I believe that prediction markets will come out looking quite good in this election. They’ve already proven their worth to me; when poll results might indicate a close or tightening race in places, the prediction market magnifies the difference, and in many cases demonstrates the poll volatility is just noise.

In the end, are the results from the prediction markets useful? Based on the number of times I’ve seen them cited this year… the answer is an unqualified YES. Are they perfect? No, and neither is any other forecasting system or technique.

I’m really looking forward to tomorrow…

The US election is just over a week away, and with that there are a few different topics I’d like to touch on. With the explosion in new prediction markets since the last presidential election, we should see some interesting (but hopefully consistent) results.

• First, a great post from Koleman Strumpf on Midas Oracle points out that half of all trades on the Betfair exchange in 2004 occurred on Election Day! While I personally think that was quite likely due to the early exit poll news for Kerry and the subsequent swing back to Bush, it proves that there are still quite a few people that may be waiting until the very last day to trade.
• Jason Ruspini just started a new thread on the Prediction Markets e-mail group regarding some divergences he’s seen between prices on InTrade and fivethirtyeight.com.

  While I think some of the things he’s observed is due to the way Nate Silver presents data on his site, Jason brings up a very good point. A thorough analysis of movement in the InTrade prediction markets should be compared to the daily calculated win percentage from fivethirtyeight (where all data comes strictly from polls). I think it could be very revealing, and give the public quite a bit more data on the accuracy of polling, aggregation of polls, and prediction markets.

• A long time ago I started four different markets on Inkling Markets that will hopefully predict control over each house of Congress, and the number of seats each party will have after the election. Data is shown here:
  
  
  (as I write this, the Democratic percentage is 53.8%, which corresponds to 234 seats in the House of Representatives.)
  
  
• I may need to update my post on prediction market software soon. Xpree (founded by Mat Fogarty, and recently joined by Leslie Fine, a well known prediction market researcher from HP) may be changing their name. The top three picks according to their contest on NameThis were:
  1. Metricast
  2. UREprojection
  3. Keymet

  Personally, I don’t like #2 or #3, but Metricast sounds interesting. It also sounds like a much better fit to what they do than “Xpree.”

  Good luck to them, if they choose to go down this route.

• Do you speak Danish? Nosco is hiring!
  For English speakers, so is InTrade and Xpree.

McKinsey & Company, the famed consulting firm, recently published a roundtable interview in its McKinsey Quarterly on prediction markets. The eleven-page article (click here to access) featured a discussion with Bo Cowgill of Google, James Surowiecki, Jeff Severts of Best Buy, and Todd Henderson (an ex-McKinsey consultant).

Some of the article had the standard “what is a prediction market” explanation. (A quick reminder that you can watch my video which explains the same thing here.) But I want to point out a number of interesting notes/quotes:

• Best Buy is rapidly catching up to Google in terms of the size of the prediction market effort. At the time of the interview, Google had run 275 markets with about 80k trades since April of 2005. Best Buy had run 147 markets with 70k trades, and involved 2,000 traders. (I believe they have been active since late 2006 or early 2007.) I wrote about the Best Buy initiative after a presentation at the ConsensusPoint New York prediction market conference. While they haven’t moved quite as fast as was mentioned there, Best Buy looks to be a strong champion of prediction markets.
• Jeff Severts of Best Buy initially started doing better forecasting with a simple survey. While not as sophisticated as a prediction market (and more time consuming to calculate averages for the forecast), it generated much better forecasts than the business had ever done before. These experiences showed that they could go further with the idea.
• Bo Cowgill at Google is clearly doing the most advanced work with the underlying prediction market data. They’re the only company that I know of that looks into the underlying organisational dynamics that prediction market data can provide. (We’ve had a bit of a blog conversation about this a few weeks back.)
• Jeff Severts also noticed that any forecasts they had done about their main competitor were not very accurate. I think this speaks to a lack of diversity amongst individuals in a company when critically examining a “competition” in which they’re involved, and similar to the optimism that Bo noted that some Google traders exhibited toward similar Google markets.
• From Bo, traders were more accurate the longer they participated, and tended to be more profitable the lower they were on the org chart.
• I intend to discuss this in more detail later, but James Surowiecki brought up that it is still an open question if prediction markets “are good at forecasting genuinely discontinuous innovations or leaps.”
• Jeff of Best Buy mentioned that “support from very senior executives is essential if you want to issue contracts on anything that might be controversial.” Particularly in traditional companies, I wholeheartedly agree. While less senior executives can make good prediction markets happen within their organisational purview, any company-wide markets need the senior executive “air cover.”
• One thing that I always try to tell people when discussing prediction markets (including in the video), is that they’re really a new way to communicate within a company. I really appreciated Jeff Severts when he said:
  

  Smartly applied, this tool can help management listen to voices, throughout the company, that otherwise go unheard.

  This is a great quote, and I believe a very wise way to think about the benefits of a prediction market.

It’s great to see a well-respected company such as McKinsey publish information on prediction markets widely. This is yet another positive step toward companies viewing prediction markets as mainstream tool, and provided some interesting insights into some of the top active prediction markets operating today.

After reading my last post, specifically this section:

The second half of the paper examined the transmission of information within Google based on the authors’ analysis of the traders and their behaviours. While there is some really interesting analysis there, it has more to do with organisational behaviour than being directly applicable to prediction markets, so I’m not going to discuss it here. But if you’re interested, I highly recommend going to read the paper!

Bo wrote the following on his blog:

This is not the first reaction along these lines. I am perplexed by the response. I can understand why other companies may not want to replicate our analysis of information flows. Perhaps it wouldn’t be worth the effort. Perhaps they would get identical results. And perhaps the company wouldn’t have the all the necessary data.

However, I expected that people could easily see value in the analysis of granular trade-by-trade data — especially if that data is joined with data about traders and outside events happening at the moment of the trades.

To which I respond here:

Perhaps this was a bit of lazy writing at the end of what felt like a longish post. But what it really came down to was my focus on the very tangible, common questions that many people I talk to have when they’re first starting with their prediction market projects. The first issue for many is just getting a market working, with sufficient participation and liquidity, and hopefully accurate predictions. These were the main points and examples from the paper that I wanted to address in my post.

I believe Bo’s concern is that with just a “first-order” discussion on the paper, we leave out the “second-order” potential. Once a market is up and running, there is a LOT of data available to the company. If you know enough about your employees (and Bo’s subsequent post suggests that any decent-sized company does) this is information you likely already have. Specifics had to be left out of the Google paper for sensitivity reasons, but the paper demonstrated the kinds of things you can discover about your organisation through a prediction market.

Addressing these questions takes time and careful attention. There is certainly a privacy issue here, but that’s more an issue of managing perceptions. (You want to make sure people trade their true beliefs, and not some altered reality because they think the corporate Big Brother is watching them.) Both the data collection and analysis would certainly take some time.

However, there is some great data and metrics available to companies that are ready to take advantage of these “second-order” advantages of prediction markets. Again, to quote Bo:

The data contains real-time metrics on the distribution of knowledge and attitudes within a firm at a highly granular level. You can get metrics on for specific of the firm, for specific classes of employees and for specific topics. You can do this for either customers or employees, and have the metrics for any moment in time. The quality of these metrics will be extremely strong, because participants have been incentivized to reveal their true expectations.

So I don’t disagree with Bo at all, I just had a slightly different focus in my discussion. Perhaps this could provoke a little further discussion on the long-term potential of prediction markets in corporations.

[UPDATE]: My sincere apologies for the spam in the original post. I have no idea what happened, but must be related to the fact I didn’t use my regular posting program to put this post up. It should be fixed now.