Mercury’s Blog

If you're new here, you may want to get Mercury's Blog by Email or subscribe to my RSS feed. Thanks for visiting!

I’m fascinated when real-money markets can be directly compared to play-money markets, particularly when I can potentially make some money.

For those of you that haven’t read my research paper, I created a series of play-money prediction markets on rowing events in the summer of 2006. The results were as what you might expect; quite accurate. When there were just 16 or more traders involved in a market, the results could be relied upon for good forecasts. I’ve turned that project into a longer-running project, which has also taken place last summer and this summer. (I’m collecting data to analyze how the number of traders required has changed since Inkling adopted Robin Hanson’s MSR, which wasn’t in place for the original research.) It’s proven quite popular amongst amateur rowers in the UK.

[I'd like to thank Inkling for providing the platform; I plan to analyze and publish the results of the last two summers' research this fall once the Olympics markets are closed out.]

A few weeks ago I created a prediction market for each of the 14 Olympic rowing events. A few days ago I was checking Betfair and realised that they had also created markets for all 14 Olympic rowing events. (Both to determine the winners as well as the podium places for each event.) When I compared the two marketplaces there were some potentially profitable discrepancies.

There seem to be two types of market-makers in the Betfair markets. One simply entered lay bets for each entrant at very poor odds (ie, 1.01 decimal) just so there was something to trade. That’s not very useful. But another type of market-maker entered more realistic odds, but odds that were dramatically skewed toward long-shots. For example, in the Men’s Double Sculls event, the play-money prediction market forecasted a win for New Zealand with about an 80% probability. However, I was able to buy shares on Betfair at 2.56, or about 39% probability! While the over-round on the market was quite large, it was because the odds on the long-shots were unreasonable. Odds on the favourites in these markets were very good, and this was the case on as many as half of the events.

So if my play-money markets are accurate (as I expect them to be), I should be able to make a little money on the Olympics, courtesy of the initial market-makers on Betfair. Some people may argue that the play-money predictions won’t be as accurate because they don’t involve real money, but looking at the current Betfair market there’s so little liquidity to have quality forecasts. Unfortunately the lack of book depth means that the mis-pricing won’t last long and won’t be hugely profitable in absolute terms, but that should be interesting to watch.

I think this is an excellent example of where a play-money market can profitably inform real-money market trading.

Questions, comments? Please enter your comments below. (No registration or sign-up required.)

(Reminder: You can subscribe to receive new posts here by e-mail, just click here.)

I read recently about an up-and-coming French company that is building what some people are calling a “prediction market.” Really, it’s just another mis-guided market in predictions.

The company is called MyPrognostic. They hope to get people on their site and make predictions. I wouldn’t classify them as a prediction market website because there is virtually no interaction past making the forecast. You can’t sell back your trades if the price rises, there’s really no way (as near as I can tell) for you to “make” money on the site via trading activity as you could on a standard futures market.

What it is, however, is an accounting system to see how good each person is at making forecasts. Once you’ve built up a reputation, you will literally be able to “sell” your forecasts to others. They believe there are customers that are willing to pay traders directly to see their forecasts.

The way MyPrognostic works is that users/traders are asked “Who will win?” a particular event/contract and provided with multiple options. They have no knowledge about percentages until they make their forecast, at which point the result is shown.

I see two major errors in this venture:

Error 1 - Finding the expert

The whole point of the Wisdom of Crowds book and general philosophy is that the crowd as a whole can consistently out-predict the expert. Certainly some experts may be better calibrated than the crowd as a whole, but it will be both very difficult to find them (due to the number of measurements required) and very difficult to stay there. How many mutual funds have consistently beat the S&P 500?

By moving away from the trading metaphor and just using a poll metaphor, I think MyPrognostic introduces some other problems. Presumably, traders will be much more likely to predict markets where there is a clear favourite, and much less likely to predict on markets where there’s little to separate the options. The problem with MyPrognostic’s model is that assuming the favourites do well, many users will look like good forecasters when all they are doing is forecasting the obvious. Presumably, a person could only trade events where the winner would traditionally have a 90% chance of winning and look like a much better forecaster than someone who was much better calibrated but traded a much wider variety of events.

I am always fascinated by businesses that start prediction markets in order to find the experts and then use their knowledge (usually to sell). The market itself is knowledgeable! You may find people that can beat the market’s consensus view in the short-term, but in the long-term few (if any) experts can maintain their position. I still think that tracking who does well in the markets (a la Google) is a great thing to do, but more to understand the dynamics of the marketplace, and not to isolate a portion of that marketplace.

Error 2 - Why would experts participate, anyway?

The only experts that I could see participating on this site are people that are so risk-averse (or lacking in capital) that they could not place the equivalent real-money bet themselves.

MyPrognostic is a little different in that you are only betting who would win a particular event, and not choosing to trade at a particular price. But at the same time, this is fairly equivalent to a typical “high street” bettor in the UK. (While some bettors examine the odds closely, others just place a bet on a particular person/team/event, no matter the odds.) If someone was a top trader on MyPrognostic they should be able to make real money by betting directly, instead of through selling their knowledge.

Summary

MyPrognostic is yet another site that is trying to leverage “The Wisdom of Crowds” to find and sell expert knowledge. I firmly believe they are mis-guided. Finding “experts” will be problematic, particularly since if they’re any good their greater incentive is to bet on a real-money site directly. I suspect they’ll figure this out in the next year or so themselves.

In my discussions with potential prediction market customers, I’ve been surprised by one particular application that seems to resonate with many executives. That is the “RAG Status.”

For those of you unaware of the acronym, it simply stands for Red-Amber-Green. It is used extensively in management “dashboard” reports to serve as a quick and efficient status marker for a project or element of a project. Instead of having to read a few sentences or paragraphs and digest them to understand the status of a project, a manager can simply glance at a colored dot.

This is the general scenario from upper managment’s perspective:

• If it’s Green, everything’s okay.
• If it’s Amber, I’m concerned and asking questions.
• If it’s Red, it’s a BIG problem and I am actively trying to solve it.

Project managers creating these reports approach it from a different perspective:

• I’ll mark it Green if I can deal with the problems.
• I’ll mark it Amber if I’m having problems and it’s likely my boss will find out about them.
• I’ll mark it Red if I’m way out of my depth and the situation is probably unsalvageable, and I’ll probably be able to blame someone else.

It is in the reputational interest of most project managers to de-emphasize the problems they’re having in order to save face and avoid unwanted upper management intrusion. However, it is in upper management’s interest to have a true accounting of the difficulties in any given project, so they can understand and properly plan for worst-case scenarios.

The result? A project starts in the Green, slides into what should should be Amber status long before it is ever reported as such, and becomes Red status before the project manager is willing to admit it. In the end the entire team (team members, project manager and upper management) wastes unnecessary time and resources fixing problems that could have been avoided at much less cost had they been identified earlier.

In my discussions on prediction markets, upper management recognise that markets are an excellent way of getting around the poor reporting incentive structure that currently exists. They know they’re being misled by project managers, but aren’t effectively able to do anything about it.

How prediction markets can be used to provide a real RAG status

The first step to providing an alternate RAG status is developing an understanding of what matters in the project. Is it meeting milestone dates? Is it meeting milestone figures, such as number of users? Is it something else entirely? This is then what the prediction market should measure.

I believe that in most cases the market should be structured in a binary, 0-100, probabilistic, “Will the project meet milestone [X]?” Employees, now incentivized to tell the truth on the actual status of the project, will generate a probability that the milestone will be achieved.

From here it is simple: instead of generated a biased Red-Amber-Green colour for a report, management will receive a 0-100% probability of success. If they specify in advance what probabilities are equivalent to the Red-Amber-Green status, prediction market forecasts will create a richer, more informational report that is still very easy for management to digest. (For example, Green = 100% - 75% forecasts, Amber = 75% - 50% forecasts, Red = 50% - 0% forecasts).

Summary

Prediction markets can remove the poor incentives between project managers and upper management. Where project managers can lie or obscure the truth with RAG status reports, when those reports are based off of prediction market forecasts the information is significantly more valuable. Upper management typically realises this, and then just needs to be sold that the costs of project overruns are much more significant than the costs of running the prediction market.

(Some of the most interesting prediction market case studies have been completed using this very technique. I would recommend watching the video of Todd Probsting of Microsoft at the Yahoo confab from back in December 2006. The streamed video can be found at this link.)