Mercury’s Blog

If you're new here, you may want to get Mercury's Blog by Email or subscribe to my RSS feed. Thanks for visiting!

In my previous post on categorizing prediction markets, one of the key differences is whether a market is public or private. (The “P” in the ICROP criteria.) There is fundamentally a very different value to the operators of a public market compared to a private market. My train of thought is below, starting with some notes on recent prediction market news.

Recent news from Hubdub

Hubdub recently announced a partnership with Reuters, which I think is a great step for public prediction markets. They previously had announced a partnership with the Huffington Post blog. When that was announced two weeks ago I was a little torn of what to think. It was great they were able to work with a major internet brand, but the implementation was pretty weak. You could find Hubdub’s markets on Huffington Post tag pages, but even knowing it was there I really had to search to find them on the page. It seemed to be something that was being treated as just a minor experiment rather than something serious.

The Reuters announcement is much more important. It looks like it’s kicking off a wider partnership program at Hubdub, which is quite exciting. Reuters has a dedicated section on the Hubdub site which will apparently be regularly updated by Reuters staff. (They’ve only created 4 questions so far.) The only scheme that I think would be better than this is if Reuters had put a dedicated Hubdub section on their (Reuters) site, and that could certainly happen down the road.

By opening up Hubdub to a wider partnership program, they will help other companies and bloggers build a reputation by allowing those people and organizations to generate interesting questions and predictions. This is great news, and should spur even more growth for both the participating bloggers and Hubdub itself.

But this got me thinking about public versus private prediction markets…

Value in Public versus Private markets

The value to operators of public markets is significantly different than private markets. I think this is why we are seeing significantly different types of growth in the two types of markets.

The value to operators of public markets comes from generating an active, thriving community of users. These users may be targets for advertising, subjects for demographic research (HSX), see examples of technology (Inkling, NewsFutures), or another unique model to be determined. This is where Hubdub looks to be pioneering.

Nigel Eccles, the CEO of Hubdub, has mentioned on many occasions some interesting statistics about an average subscriber’s interaction with a newspaper’s online presence. For most newspapers it is extremely limited; a person will check a story or two and leave. There is little real engagement, and so page views and advertising rates aren’t as high as they could be. (Alex Kirtland talked about this here.) However, prediction markets have shown themselves to be hugely engaging, and can also be made highly local and relevant to a small audience. This looks to be the way they’re going with their partnership program. Each partnership will add and build another sub-community of users, which adds value to both the partner and Hubdub.

The value is significantly different for private markets. When I think of private markets, I think of a corporate prediction market where the company is looking to get useful and accurate business intelligence from their employees. The business intelligence is the value many companies claim they are trying to capture.

The difficulty in private markets is that there is no obvious, traceable value chain. In that I mean that most companies cannot say that because the market told management that there was X% chance of event A occurring, the company changed strategy and saved $Y. Many companies are in reality treating them as non-actionable market intelligence, where they examine only after the fact how accurate the predictions were.

Even if a company did trust their employees enough to take action directly based off of what their internal markets were telling them, it may still be difficult to calculate the value of that intelligence. Particularly since management wants to be (or at least appear) smarter than their employees, it is quite easy to claim after the fact that they would have taken the same actions based off of other intelligence.

Fundamentally, it takes a company that both trusts their employees enough to take action on the market indicators and management that is honest enough in what would have happened without that intelligence in order to calculate the value of those prediction markets. For example, Mat Fogarty has talked about how he used prediction markets at EA to quantify game quality scores, which is certainly useful. But where that can directly turn into additional profit is if EA (or any similar company) took prediction market intelligence to adjust how they filled their distribution channels. They could save money by not creating unnecessary copies of bad games, and could make more money by ensuring they had enough copies of hit games ready when they went on sale. As far as I am aware, few companies have taken that final step to action based on market results.

Prediction markets certainly add value even where the elements I mentioned above aren’t present, it’s just that the direct value cannot be easily calculated. And until an executive can directly point at how the cost of a prediction market is more than made up by the direct value added, the growth of prediction markets will be limited.

That said…

Prediction markets are clearly a growth industry, even in private markets. Inkling Markets, NewsFutures, and Xpree have all recently hired great new people into their businesses, so the market for private markets is clearly growing. But until the value calculation above can be directly made, that growth rate just isn’t as high as I wish it would be.

(Gratuitous photo is from a recent holiday, specifically a section of the Great Wall of China outside of Beijing.)

I’ve had a couple of discussions recently about the general state of prediction markets; where the industry is going, where companies are finding successes and growth, etc. Some of these discussions can be difficult since there are several dimensions to a prediction marketplace, and thus the dynamics of what makes each successful (or not) varies.

So I am proposing a categorization method for prediction markets. Together, the criteria should be exhaustive and mutually exclusive. This could hopefully help describe the wide variety of potential prediction marketplaces.

Without further ado, here it is:

ICROP Criteria

• [I] - Interest: Horizontal, Vertical

  Horizontal indicates a broad focus, such as the exchanges run by Inkling, Newsfutures, and Hubdub. Vertical prediction markets are focused on a particular industry or sector, such as the Hollywood Stock Exchange, the now-defunct Storage Markets, or many enterprise prediction markets.

• [C] - Currency: Real-Money, Play-Money

  This should be straightforward. Is real-money or play-money used to trade on the marketplace?

• [R] - Rewards (maximum): Real (tangible) rewards, Intangible rewards

  Rewards should be classified by the maximum reward available on the exchange. It only distinguishes between tangible rewards, such as cash or gifts, and intangible rewards such as names on a leaderboard. So even Newsfutures and Hollywood Stock Exchange would be classified as Real Rewards sites, since you can win prizes on both exchanges. This is a fundamentally different potential motivation than a site like Inkling or Hubdub, where there are no tangible rewards available at all.

• [O] - Objective: Community, Market Results, or Background data

  This refers to the objective of the market creator. Is the purpose of the site to build or leverage a community, such as Inkling or the Chicago Sun-Times?

  Is the purpose to generate unique market results, such as Intrade or an enterprise prediction market?

  Or is the purpose to gather both the market results and information about the traders to generate additional background data known only to administrators, such as HSX?

• [P] - Privacy: Private, Semi-Private, or Public

  Again, this is largely straightforward. The semi-private case accounts for sites like Betfair that is generally public, but needs to limit participation for legal reasons.

Examples

These are how I would classify a number of prediction marketplaces based on the ICROP criteria.

Betfair:
I - Horizontal
C - Real-Money
R - Real rewards
O - Market results
P - Semi-Private

HSX:
I - Vertical
C - Play-Money
R - Real rewards
O - Background data
P - Public

Hubdub:
I - Horizontal
C - Play-Money
R - Intangible rewards
O - Community
P - Public

InTrade:
I - Horizontal
C - Real-Money
R - Real rewards
O - Market results
P - Semi-Private

Newsfutures:
I - Horizontal
C - Play-Money
R - Real rewards
O - Community
P - Public

[Acme] Enterprise prediction market:
I - Vertical
C - Play-Money
R - Real rewards
O - Market results
P - Private

Some initial thoughts

A few things strike me immediately. First is how so very few prediction marketplaces really utilize what I’m calling “Background data.” (This is something that Bo has mentioned before in reference to his prediction market work at Google.) There is a tremendous opportunity both in public and private markets to analyze and use trader data such as demographic information in conjunction with market data to create a completely new dataset. The only public site that I know of that has really done well here is the Hollywood Stock Exchange. But like Bo, I believe that this information could potentially be invaluable for corporate prediction markets. Unfortunately, none of the software solutions available have good feature-sets for this.

Second, there is certainly a link between Currency and Rewards. Real-money markets result in real tangible rewards, where play-money markets can result in either tangible or intangible rewards. Perhaps they shouldn’t be somehow combined, but I think they are important enough to list separately.

Summary

This isn’t a fully-formed classification scheme, and would certainly appreciate any comments (publicly in the comments or privately via e-mail) to refine it.

I read recently about an up-and-coming French company that is building what some people are calling a “prediction market.” Really, it’s just another mis-guided market in predictions.

The company is called MyPrognostic. They hope to get people on their site and make predictions. I wouldn’t classify them as a prediction market website because there is virtually no interaction past making the forecast. You can’t sell back your trades if the price rises, there’s really no way (as near as I can tell) for you to “make” money on the site via trading activity as you could on a standard futures market.

What it is, however, is an accounting system to see how good each person is at making forecasts. Once you’ve built up a reputation, you will literally be able to “sell” your forecasts to others. They believe there are customers that are willing to pay traders directly to see their forecasts.

The way MyPrognostic works is that users/traders are asked “Who will win?” a particular event/contract and provided with multiple options. They have no knowledge about percentages until they make their forecast, at which point the result is shown.

I see two major errors in this venture:

Error 1 - Finding the expert

The whole point of the Wisdom of Crowds book and general philosophy is that the crowd as a whole can consistently out-predict the expert. Certainly some experts may be better calibrated than the crowd as a whole, but it will be both very difficult to find them (due to the number of measurements required) and very difficult to stay there. How many mutual funds have consistently beat the S&P 500?

By moving away from the trading metaphor and just using a poll metaphor, I think MyPrognostic introduces some other problems. Presumably, traders will be much more likely to predict markets where there is a clear favourite, and much less likely to predict on markets where there’s little to separate the options. The problem with MyPrognostic’s model is that assuming the favourites do well, many users will look like good forecasters when all they are doing is forecasting the obvious. Presumably, a person could only trade events where the winner would traditionally have a 90% chance of winning and look like a much better forecaster than someone who was much better calibrated but traded a much wider variety of events.

I am always fascinated by businesses that start prediction markets in order to find the experts and then use their knowledge (usually to sell). The market itself is knowledgeable! You may find people that can beat the market’s consensus view in the short-term, but in the long-term few (if any) experts can maintain their position. I still think that tracking who does well in the markets (a la Google) is a great thing to do, but more to understand the dynamics of the marketplace, and not to isolate a portion of that marketplace.

Error 2 - Why would experts participate, anyway?

The only experts that I could see participating on this site are people that are so risk-averse (or lacking in capital) that they could not place the equivalent real-money bet themselves.

MyPrognostic is a little different in that you are only betting who would win a particular event, and not choosing to trade at a particular price. But at the same time, this is fairly equivalent to a typical “high street” bettor in the UK. (While some bettors examine the odds closely, others just place a bet on a particular person/team/event, no matter the odds.) If someone was a top trader on MyPrognostic they should be able to make real money by betting directly, instead of through selling their knowledge.

Summary

MyPrognostic is yet another site that is trying to leverage “The Wisdom of Crowds” to find and sell expert knowledge. I firmly believe they are mis-guided. Finding “experts” will be problematic, particularly since if they’re any good their greater incentive is to bet on a real-money site directly. I suspect they’ll figure this out in the next year or so themselves.

In my discussions with potential prediction market customers, I’ve been surprised by one particular application that seems to resonate with many executives. That is the “RAG Status.”

For those of you unaware of the acronym, it simply stands for Red-Amber-Green. It is used extensively in management “dashboard” reports to serve as a quick and efficient status marker for a project or element of a project. Instead of having to read a few sentences or paragraphs and digest them to understand the status of a project, a manager can simply glance at a colored dot.

This is the general scenario from upper managment’s perspective:

• If it’s Green, everything’s okay.
• If it’s Amber, I’m concerned and asking questions.
• If it’s Red, it’s a BIG problem and I am actively trying to solve it.

Project managers creating these reports approach it from a different perspective:

• I’ll mark it Green if I can deal with the problems.
• I’ll mark it Amber if I’m having problems and it’s likely my boss will find out about them.
• I’ll mark it Red if I’m way out of my depth and the situation is probably unsalvageable, and I’ll probably be able to blame someone else.

It is in the reputational interest of most project managers to de-emphasize the problems they’re having in order to save face and avoid unwanted upper management intrusion. However, it is in upper management’s interest to have a true accounting of the difficulties in any given project, so they can understand and properly plan for worst-case scenarios.

The result? A project starts in the Green, slides into what should should be Amber status long before it is ever reported as such, and becomes Red status before the project manager is willing to admit it. In the end the entire team (team members, project manager and upper management) wastes unnecessary time and resources fixing problems that could have been avoided at much less cost had they been identified earlier.

In my discussions on prediction markets, upper management recognise that markets are an excellent way of getting around the poor reporting incentive structure that currently exists. They know they’re being misled by project managers, but aren’t effectively able to do anything about it.

How prediction markets can be used to provide a real RAG status

The first step to providing an alternate RAG status is developing an understanding of what matters in the project. Is it meeting milestone dates? Is it meeting milestone figures, such as number of users? Is it something else entirely? This is then what the prediction market should measure.

I believe that in most cases the market should be structured in a binary, 0-100, probabilistic, “Will the project meet milestone [X]?” Employees, now incentivized to tell the truth on the actual status of the project, will generate a probability that the milestone will be achieved.

From here it is simple: instead of generated a biased Red-Amber-Green colour for a report, management will receive a 0-100% probability of success. If they specify in advance what probabilities are equivalent to the Red-Amber-Green status, prediction market forecasts will create a richer, more informational report that is still very easy for management to digest. (For example, Green = 100% - 75% forecasts, Amber = 75% - 50% forecasts, Red = 50% - 0% forecasts).

Summary

Prediction markets can remove the poor incentives between project managers and upper management. Where project managers can lie or obscure the truth with RAG status reports, when those reports are based off of prediction market forecasts the information is significantly more valuable. Upper management typically realises this, and then just needs to be sold that the costs of project overruns are much more significant than the costs of running the prediction market.

(Some of the most interesting prediction market case studies have been completed using this very technique. I would recommend watching the video of Todd Probsting of Microsoft at the Yahoo confab from back in December 2006. The streamed video can be found at this link.)