Mercury's Blog

There has recently been quite a bit of discussion on MidasOracle over the value of prediction markets. Part of this was sparked off by the recent Economist article on prediction markets, but part of it has to do with things Chris Masse has been writing for some time now. Chris is very vocal about his views, and I wanted to put a different perspective forward here.

Economist article

A recent article on prediction markets in the Economist was titled “An uncertain future.” I would describe its attitude toward prediction markets as mixed. Here are a few selected quotes:

“But although they have spread beyond early-adopting companies in the technology industry, they have still not become mainstream management tools. Even fervent advocates admit much remains to be done to convince sceptical managers of their value.”

“Koch says the results so far have been pretty accurate compared to actual outcomes, but stresses that markets are complementary to other forecasting techniques, not a substitute for them.”

“Another reason prediction markets flop is that employees cannot see how the results are used, so they lose interest. [...] its most effective trials took place in areas where managers could do something with their findings, making staff feel that trading was worthwhile.”

“Bosses may also be wary of relying on the judgments of non-experts. Yet many pilot projects run so far have shown that junior staff can often be surprisingly good forecasters.”

So I believe the article is generally positive toward the possibility of prediction markets, but generally negative toward the lack of adoption. Which is fair; though I think some good success stories have been kept quiet by some companies for competitive advantage and corporate ego.

Chris Masse’s reaction

Chris Masse has been writing about the prediction market industry for a number of years, and started his blog MidasOracle in about 2006. Strangely, Chris’ writings on prediction markets have become quite negative in recent months. As I understand it, he’s become quite irritated that prediction markets have been touted as “vastly superior” to polls in elections, and other general marketing in the industry. The front page of MidasOracle now has the following statement (his emphasis removed):

The Truth about Prediction Markets

The social utility of the prediction markets is marginal. Number one, the aggregated information has value only for the totally uninformed people (a group that comprises those who overly obsess with prediction markets and have a narrow cultural universe). Number two, the added accuracy (if any) is minute, and, anyway, doesn’t fill up the gap between expectations and omniscience (which is how people judge forecasters). In our view, the social utility of the prediction markets lays in efficiency, not in accuracy. In complicated situations, the prediction markets integrate expectations (informed by facts and expertise) much faster than the mass media do. Their accuracy/efficiency is their uniqueness. It is their velocity that we should put to work.

Remember, dear readers, you heard it here first —on Midas Oracle.

Part of my issue is his definition of “social utility”: namely that I’m not exactly sure what he’s talking about. I completely understand the concept of utility, but “social utility” is a phrase that could mean a variety of things, and I don’t know exactly what Chris means when he writes about it.

My other response to Chris is that I feel he doesn’t distinguish between public and internal prediction markets when he discusses marketing and usage of these markets. I’d like to discuss these issues here.

Public prediction markets

Public prediction markets are tricky things. They’re tricky to get a critical mass of people involved, they’re tricky to get a flow of interesting and valuable contracts, and they’re tricky when events happen that don’t easily conform to how the contracts were established. That said, there are a handful of good and interesting public PM’s. InTrade and Betfair both offer real-money markets, and the better play-money markets include Hubdub, Inkling, Newsfutures, and HSX.

These markets do get press, particularly InTrade in the run-up to the last two presidential elections. InTrade in particular seems to be irritating Chris because of the data point that they forecasted all 50 states correctly. (Which as I’ve written before probably means that the markets weren’t running as efficiently as they could; you’re supposed to get market “failures”.) And I agree with him that too much has been made of this data point.

Overall, I think Chris rightly addresses the issue of the value of public prediction markets. In many cases they’re purely entertainment, and in other cases (even the elections), they’re not necessarily better predictors than other methods. That said, they do incorporate new information faster than any other forecasting model. But I believe Chris wrongly applies these same criticisms to a completely different model, internal prediction markets.

Internal (corporate) prediction markets

Internal prediction markets seek to do either of two things:

• forecast something important to the company where they already have a prediction
• forecast something that’s never been predicted before

When it comes to the first point, forecasting something that the company already forecasts, prediction markets may or may not be an excellent solution. I’ve seen one set of markets that absolutely blew away the accuracy of current forecasts, and I’ve seen other markets that were consistent with current forecasts with little or no accuracy edge. In this case, prediction markets can serve as a fairly low-cost “reality check” on the official forecasts. When they deviate too much, it will raise a flag for investigation. This deviation is particularly helpful since prediction market forecasts are real-time, and can react instantly to new news.

I think the second point above is perhaps the most valuable opportunity. Simple ideas like using a prediction market to establish a RAG status could potentially be very powerful in a company. Depending on the value of the information forecasted, even a moderately accurate prediction market could be incredibly useful to a company.

Chris also over-reacts to some of the marketing from software vendors. Yes, some may mention events like InTrade predicting all 50 states in 2004, but that’s just a party trick to get people interested and in the door. Companies first need to be convinced that the tool works, then they need to be convinced that it works for them. The second part is done individually with their clients so we don’t see it. (Though I really like the case studies that Inkling have put on their website.)

I personally think that he gives too much credit to the popular press in raising the profile of prediction markets; far more credit is due to James Surowiecki and “The Wisdom of Crowds.” (Amazon link)

Summary

Prediction markets are interesting tools, but the lessons learned from public prediction markets are different than those learned from internal prediction markets. It’s important that these two applications are not confused. There will always (and rightly) be questions about the accuracy of prediction markets. In some cases it’s clear that markets are superior, in some internal cases it doesn’t matter since nothing is forecasted right now, and in other cases markets will be about as accurate as what’s already predicted. But in all three cases there is still a valuable argument as to why prediction markets should be used. It all comes down to the specific needs of an industry or company, which is where the vendors step in to help.

The results speak for themselves. Each year there are more software vendors, and each year the existing software vendors hire more people to serve their clients. I’m not going to try and predict the long-term future, but I believe the short-term future is positive.

Happy New Year to everyone! I hope that 2009 is happy, healthy and prosperous for you. I specifically wanted to thank you as a reader of this blog, and those who have linked to my posts here. Google recently refreshed their PageRank for Mercury’s Blog, and it now has a PageRank of 6/10! Though I don’t post often, I do try to write long-form news and analysis, and I’m glad you’ve found it useful.

What I expect for prediction markets in 2009

I’m probably going to regret doing this by the end of the year, but I’m going to put on my forecasting hat and try to predict what will happen this year in the field of prediction markets.

• Prediction markets in 2009 are going to become even more well-known and wide-spread, but there will be no single event that brings them to the attention of the public. It’s going to be a slow, but steady, growth.
• All of the prediction market vendors will mature their business offering/proposition. Both InklingMarkets and Xpree have made recent business development hires, and Inkling recently redesigned their website with a stronger business pitch and case studies. I think that business cases for prediction markets will be more clear and more developed in general because of this.
• HubDub will continue to only be the only strongly popular play-money prediction market. (HSX is probably similarly popular, but is a single-industry prediction market.) With Hubdub’s partner program and social networking features, they will see significant growth this year.
• While a couple additional software vendors may appear, I get the feeling that the market for prediction market software is largely saturated. People working on projects part-time and selling them may proliferate and be suitable for the lower end of the market, I don’t see many more serious prediction market software companies starting up in 2009.
• I’m looking forward to see how the CantorExchange develops. Will there be enough customers? Will the market see the effects of other studios trying to bid the prices of their competitors’ films down? Or will it be seen more as a financial hedging tool for institutions involved in the film industry? There are lots of open questions to evaluate here, but I still maintain it’s a great step forward for the prediction markets industry.

What to expect from this blog in 2009

I’m going to continue posting on both prediction markets, but continue talking more and more about how collective intelligence can be used in other contexts. Specifically, I’ve become really interested in how groups of people develop new ideas and new innovations.

As I’ve written before, I really don’t like the word “innovation.” Bruce Nussbaum at BusinessWeek has written a few things lately that I really like:

“Innovation” died in 2008, killed off by overuse, misuse, narrowness, incrementalism and failure to evolve. It was done in by CEOs, consultants, marketeers, advertisers and business journalists who degraded and devalued the idea by conflating it with change, technology, design, globalization, trendiness, and anything “new.” It was done it by an obsession with measurement, metrics and math and a demand for predictability in an unpredictable world.

and

“Innovation” is inadequate as a concept to deal with these changes. You have “game-changing” innovation, which is big but rare and incremental innovation which is small but common.

The truth about innovation is that it takes groups of people a lot of work to find and develop new ideas and turn them into innovations. Scott Berkun has a great chapter in his book “The Myths of Innovation” where he destroys the myth of the lone creative genius.

We know it takes groups of people to make innovation happen, so where are the tools for it? A lot of time, money and effort has gone into social networking software to connect people, such as Facebook, MySpace, etc. But I have to agree with Tim O’Reilly: we need to stop throwing sheep and do something worthy. But those tools can be very useful to us in helping foster innovation. I reviewed the software for idea and innovation software recently, but was left generally unimpressed.

These are some of my current thoughts. I plan on thinking out loud more in 2009 on these topics and seeing where it takes me. I look forward to your feedback.

Are you a successful trader on the Hollywood Stock Exchange (aka HSX)?

Well, now you could potentially turn that expertise into cold, hard cash. Cantor Fitzgerald, the company that owns HSX, has announced a real-money equivalent, the Cantor Exchange. On it you’ll be able to trade “Movie Box Office Contracts.”

It was officially announced Monday, December 8th through press releases on the new CantorExchange website. To quote from the press releases:

Cantor Fitzgerald, L.P., a leading global financial services firm, announced today that it has filed an application with the Commodity Futures Trading Commission (“CFTC”) to launch the Cantor Exchange. Cantor Exchange intends to list Domestic Box Office Receipt contracts as the exchange’s first traded product.

[...]

Subject to final regulatory approval of the Cantor Exchange application, Domestic Box Office Receipt contracts will offer film finance professionals and traders a new opportunity to hedge and speculate on the theatrical performance (ticket sales) of major film titles. Domestic Box Office Receipt contracts will be a next generation financial management tool that allows film professionals to hedge risk and provides speculative opportunities to other market participants based on the first four weeks of a film’s box office performance.

The first Domestic Box Office Receipt contract is expected to be listed on the Cantor Exchange in the first quarter of 2009, subject to final approval of the Cantor Exchange application by the CFTC.

If you’re willing to sit through the video’s cheesy music on the site, you’ll find out that traders can get started with $50 in their account. It appears they’re pitching the buy-side (generally) for fans looking to share in the up-side and pitching the sell-side (generally) for investors and others looking to hedge their investment.

Each contract starts with a week-long auction to determine the starting price. Every 24 hours the exchange will publish the overall market price that would be achieved based on all the buy and sell orders entered. Before going to continuous trading, all buy/sell orders will be cleared at the single best market price.

Summary

I think this is great news, as it starts to push the boundaries of acceptance of futures (aka “prediction”) markets. I wish Alex and the rest of the team there all the best luck in the approval process and look forward to trading a bit there myself!

It should be interesting to hear what the mainstream press thinks of this development.

So a number of interesting things have come across my desk recently, all of which I wanted to write about but none of which really became its own post.

Askmarkets.com has launched!

George Tziralis‘s new project, Askmarkets, has now officially launched. George has been beta-testing it for a while, so it’s great to see it officially go live.

The most unique thing about Askmarkets to me is that it uses David Pennock’s DPM (Dynamic Pari-Mutuel Market) as an automatic market-maker instead of Robin Hanson’s MSR (Market Scoring Rule). It largely operates the same way, though with a DPM the person/company running the market isn’t required to subsidize the market-maker losses. (This theoretically makes it more attractive for eventual real-money applications.)

Nostradamical.com has launched

This is a new prediction market site that I heard about recently. It seems like a decent site, and I really like that it was developed by just one guy (Brad Young) here in the UK! While you can show your confidence in a particular outcome through a slider, I don’t like that I can’t adjust how much I “bet” on a given outcome. While I might be very confident about something in my industry, I wouldn’t want to bet the same amount on a pop culture topic. (I’m likely the last to know any Britney Spears news.)

Good luck to Brad as he improves and expands the site!

CFTC new scope?

The CFTC may be gaining a much broader new scope. If Senator Harkin passes his bill, it “would force all over-the-counter derivatives, including credit-default swaps, onto regulated futures exchanges.” This would be overseen by the CFTC.

To quote some interesting lines from the Wall Street Journal article:

[...]those financial contracts would be standardized and centrally cleared. It would also mean that they could be subject to CFTC-imposed speculative position limits and federal reporting requirements.

Under current federal law, certain financial derivatives are excluded from the definition of a commodity. The law, which was amended in 2000, created distinctions between what are known as “exempt” and “excluded” commodities.

The CFTC has no regulatory authority over excluded commodities, which include credit-default swaps. Transactions involving exempt commodities, such as metals and energy, may not fall under some of the CFTC’s rules and regulations if those contracts meet certain requirements.

Sen. Harkin’s new bill, dubbed the Derivatives Trading Integrity Act, would eliminate the distinctions in the law between excluded and exempt commodities, treating them all the same.

Hedging versus Gambling

Speaking of the CFTC, I was talking with some people recently about how to define the difference (in more layman’s terms) between hedging and gambling. The central question was: is there an inherent risk?

The classic case is that of farmers having a risk of their crop prices changing dramatically during the growing & selling seasons. The Chicago Board of Trade helped farmers hedge their inherent risk of significant market changes between the time they planted their crops and when those crops could be sold.

That can be contrasted to sports, which are typically understood as gambling. While some individuals may have emotional risk tied up in a teams’ win or loss, there is no inherent risk… sports are entertainment first and economic enterprises second.

In the middle, however, are contracts on something like the US Presidential election. A change in government ripples across the economy, and so businesses and individuals may very well have some risk in various election outcomes.

So while the question “is there an inherent risk?” helps to clarify some situations, there is still a spectrum of answers. While this may be second nature to some or many readers, I think it’s a better question to help separate hedging from gambling in many novice/lay-person’s minds.

Google Prediction Markets case at HBS

Andrew McAfee is a professor at Harvard Business School who does a lot of work, research and teaching on “Enterprise 2.0″. He has worked with/talked to Bo Cowgill at Google and has written a case study on Google’s prediction markets.

He’s also a twitter user and I wanted to quote one of his “tweets” here:

Google prediction markets case teaches like a dream – http://snurl.com/5q78q

I personally think this is a really good, solid step towards recognizing prediction markets as a valid and valuable internal tool. Hopefully more and more people will come into business knowing that prediction markets exist and that they can provide valuable information to managers about what their company is doing.

My HubDub post-election market

Finally, I put up my own post-election market on HubDub, on US Supreme Court justices. Please check it out below and trade!

How many US Supreme Court Justices will retire by the end of 2009?

There is a huge mis-conception in the media when it comes to evaluating the success of prediction markets in the recent US election. Simply put, you have to be wrong to be right. But depending on the type of prediction market you are operating, different methods of assessment are required. I’m going to go through each type of market and compare it to the best-of-class of poll aggregators (fivethirtyeight.com), discuss what they came up with, and potentially find a potential winner from the 2008 forecasts.

Probabilistic predictions (and prediction markets)

The most popular prediction markets this election season were probabilistic markets, where the payoff was either 1 or 0. (In the case of InTrade, $10 or $0.) The market price can and is interpreted as a percent probability that the contract will take place; one of the candidates getting elected. Unfortunately, many commentators believe that once a contract is above 50% that the candidate will win, and if a candidate is above 50% in a given market and doesn’t win, the prediction market has failed.

This is completely wrong.

I’ve discussed in previous posts that you have to be wrong in order to be right. As a quick reminder:

• 98% chance = 1 in 50 will be wrong
• 90% chance = 1 in 10 will be wrong
• 80% chance = 1 in 5 will be wrong
• 75% chance = 1 in 4 will be wrong
• 67% chance = 1 in 3 will be wrong
• 50% chance = 1 in 2 will be wrong

So how did the prediction markets do? And how did fivethirtyeight.com do in comparison? (They’re the only site I know of that takes poll results and turns them into probabilistic forecasts mathematically.) Here’s a quick snapshot:

InTrade results

We immediately run into the primary problem with evaluating these markets… while there are 50+ markets, only a small number of these were not at the extremes of the scales. I’ve just included those with a market price of <90%.

This looks great for InTrade, but it's deceiving. For example, let's take a look at all contracts at approximately 80%. (FL, OH, VA, NV, and CO). They have an average market price of 82.6%. If InTrade was perfectly calibrated, one of these should have been wrong.

It looks better on the other side. If you calculate MT, GA, ND, and IN they have an average market price of 26.75%. In fact, one of these four were wrong (Indiana).

For the mathematical amongst you, it should be clear that this type of assessment is quite crude. When you have so few data points, you need to pick and choose "bins" of data with your own best judgement. Ideally there would be enough markets that you could create strict "bins" of data and measure against those. (This is exactly how Inkling created their plots here… I would encourage you to read their post, too.) Unfortunately, we only have a small number of “battleground” markets and this just isn’t possible.

Fivethirtyeight.com results

In my opinion, fivethirtyeight.com appears to have done marginally better. Take the bottom five markets shown, which have a combined probability of 15%. In fact, one of these five actually occurred; a 20% success rate.

Another slice of data shows the same thing: MT, ND, IN, and MO have an average probability of 30% and an actual success rate of 25%. Taking MO, NC and FL the probability was 61%, and and actual success rate of 67%.

I say that fivethirtyeight.com has done marginally better because I could take reasonable slices of data throughout their predictions and come up with reasonably calibrated results. I had to specifically pick and choose to find similar results with InTrade. In other words, the fivethirtyeight.com forecasts were more internally consistent.

Non-probability predictions (and prediction markets)

Some of the lesser-cited and more lightly-traded prediction markets for the 2008 election cycle were on index markets on the vote share for each candidate. Here was the final tally of the national vote:

• Democrat – Obama – 52.7%
• Republican – McCain – 46.0%
• Other – 1.3%

What did the Iowa Electronic Market forecast? (Data from midnight on the 3rd)

• Democrat – Obama – 53.5% (0.8% error)
• Republican – McCain – 46.4% (0.4% error)

What did fivethirtyeight.com forecast?

• Democrat – Obama – 52.3% (0.4% error)
• Republican – McCain – 46.2% (0.2% error)
• Other – 1.5% (0.2% error)

To be fair, fivethirtyeight.com was the best of the “poll aggregators”. Real Clear Politics and Pollster.com came up with the following:

• Democrat – Obama – 52.1%, 52.0%
• Republican – McCain – 44.5%, 44.4%

It’s clear from this data that while the Iowa Electronic Markets were quite accurate, fivethirtyeight.com forecasts were even more accurate.

For reference, redbluerichpoor.com showed the following result from fivethirtyeight.com’s final forecasts (of state vote share) which were pretty accurate:

The issue of time

Recently, George Neumann of the Iowa Electronic Markets sent out a document to the Prediction Markets Google group that claimed that the IEM “continue to dominate polls.” One specific line really struck me as ridiculous:

During this 886 day period the average absolute error was 1.2%, amazingly similar to the final polling results but for a much longer period.

So now we’re supposed to assess the accuracy of a prediction market over a 2+ year period?!? So as the race moves and swings, and the markets with it, the IEM seems only to be concerned about the average. This implies that the election is much more static than I believe it to be.

Face it, the race changes. If the election was held within a week of the Republican convention, John McCain would have likely won… the markets reflected that. But events change, and the market changes along with them. This is why I disagree with Peter McClusky’s analysis of price changes here.

Uncertainty is priced into prediction market prices; in the fall Nate Silver of fivethirtyeight.com was consistently predicting a much higher probability of Obama winning the race than prediction markets, because the uncertainty of events between the forecast date and the election was priced into the market. As the election date got closer the uncertainty was removed from the price and Obama’s price went up. But this happens rather late in election futures; in election after election we’ve seen examples of late-breaking news that has had the ability to shift the outcome of a race.

Where does this leave prediction markets?

Were prediction markets consistently better than a well-performing site like fivethirtyeight.com? No.

Were prediction markets consistently worse than a site like fivethirtyeight.com? No.

They perform largely the same, though the final accuracy of fivethirtyeight.com was a tad bit better than InTrade/IEM. But the purpose of the two types of sites are different.

Fivethirtyeight.com and similar sites take current data and process it to extrapolate trends. These sites lag real events; Nate Silver mentioned on a number of times that he expected his model’s forecast to move, but that it hadn’t because the relevant polls hadn’t hit the model yet. Between the time a poll closes, the result released and then incorporated into the model is anything from a day or two to several days. So while it looks quite accurate, a poll aggregator is a lagging indicator. (Another couple of election cycles will tell us if their accuracy continues or was just a fluke in 2008.)

Prediction markets show the results of what a group of traders believe what will happen. This includes polling data, but also reacts to real-time information. A candidate makes a huge gaffe, and the market price will reflect it in minutes, where a poll aggregator could take days to see any effect.

This is the “social utility” of prediction markets, to answer a question that Chris Masse always poses. While they are on par with the accuracy of the best poll aggregators, their forecasts are real-time and reflect the state of the race right now. No other mechanism does this. While markets are certainly fed by polls, that isn’t the whole puzzle in and of itself.

Prediction markets worked quite well again this election cycle. Though their final forecasts were on par with the best poll aggregators, their real-time forecasts throughout the election season is the reason why they should be examined and discussed more broadly.