Mercury's Blog

“In God We Trust; all others must bring data.”

– W. Edwards Deming

There has been more discussion recently on Midas Oracle, here and other blogs about the value of enterprise prediction markets. Part of this is because we don’t have a good measurement of value, and partly because a lack of information.

When it comes to value, there’s one great way to determine if a prediction market is valued by a company: they keep using it! No matter what people “think” the value of an enterprise prediction market may be, if the actual customer is willing to pay for it, the tool is valued.

But another big part of why we’re even having this discussion is because there’s no clear perspective on what’s going on with the industry. Why? Think of the parable of the blind men and the elephant. (Quote from Wikipedia).

A group of blind men (or men in the dark) touch an elephant to learn what it is like. Each one touches a different part, but only one part, such as the side or the tusk. They then compare notes on what they felt, and learn they are in complete disagreement. The story is used to indicate that reality may be viewed differently depending upon one’s perspective, suggesting that what seems an absolute truth may be relative due to the deceptive nature of half-truths.

So the problem is, we’re all talking our of our collective a**es.

There’s no way we can talk intelligently as a community unless we have a shared understanding of what’s actually going on in the industry.

The measurement

I propose that we start by looking at two measurements: retention rate and customer growth rate. Retention rate measures how long a customer stays active. This should be a relatively good indication of the value a prediction market provides to a client; the longer they pay for it the more valuable it is to them!

Customer growth rate is exactly that, how quickly the industry is growing and finding new clients.

While rates won’t provide a total magnitude on the size of the industry, it should provide a good proxy of value to customers and growth of the customer base. It’s not the sum-total of what can be measured right now, but I think it’s a solid first step.

A proposal

I hereby make a public proposal. In order that the entire industry can talk about a common set of data, I volunteer to act as the point of contact for data aggregation. Enterprise prediction market vendors would only need to provide limited data, and would get in return a comparison of their own company’s statistics to the industry at large. The public would get the aggregated statistics (only, no company-specific details) on the retention rate and growth rate of the industry.

The information that would be required of the Enterprise Prediction Markets software vendors is:

A list of clients (who don’t have to be named, code names/numbers okay) and for each of those:

• Date (by quarter) when the client was first invoiced
• Date (by quarter) when the client was last invoiced

I believe that invoicing is a good proxy for measuring the start and end of a prediction market, but if a software vendor would like to use other measures that’s fine, as long as they fairly represent the start and end of a paid prediction marketplace.

Thoughts?

I’d be interested in your thoughts on this. So far I’ve gotten some solid interest from some of the software vendors, but am posting this here to gather some additional interest. (Yes, a little public pressure.)

I think that if we’re confident enough in the prediction market industry, we shouldn’t be afraid of the data!

PS- Because of a recent spate of comment spammers, I’m moderating all comments so you see a bit of a delay before your comment shows up here. But once I approve it, you won’t need to wait if/when you comment again.

I’ve had a number of comments to my recent post on prediction markets and how they approach forecasting differently than other mechanisms, both here and on MidasOracle. I’d like to respond to a number of these comments here.

Comments and Criticism

[Chris Masse]: “Number one, I don’t understand why information aggregation would be a “bottom-up” approach (as opposed to “top-down”). Our traders bring bits of information to the market —but these bits of information were originally produced by the traditional sources (news, political polls, political forecasters, opinion leaders, etc.). I don’t understand why this “bottom-up” metaphor would apply to the prediction markets.”

I have a few comments for this. First, I don’t know where Chris gets the idea that “bits of information were originally produced by the traditional sources.” Corporations aren’t trading on political markets where there are polls and expert opinions. They’re trading on things that matter to their company and to their industry. While what he mentions certainly applies to public prediction markets, it’s virtually irrelevant to corporate markets. Individual traders bring their judgement and the perspective from their place in the company and their personal history, which when combined with other employees in the company is very valuable indeed.

With the “bottom-up” metaphor, I was trying to show that forecasts are built from the views and opinions of individual employees… from the bottom-up. “Top-down” is how much of traditional forecasting is done: put data through a model at corporate HQ and generate a forecast which is then distributed throughout the company… from the top-down.

[Chris Masse]: “[trying to paraphrase me] EPMs are such a novelty, and the corporate forecasters such a bunch of retarded people, that it will take decades before commercial organizations get to adopt the prediction market tool.”

It’s not that corporate forecasters are retarded, just that prediction markets are completely different to anything they’ve ever encountered for forecasting. And like anything different, they’re generally going to be ignored. Note that virtually all of the corporate prediction market trials are NOT initiated by forecasters, they’re initiated by general managers who aren’t so directly tied into a specific forecasting tool world-view.

[Chris Masse]: “If enterprise prediction markets were such a revolutionary and powerful forecasting tool, it would have found a market already —just like the iPod, the iPhone, FaceBook or Twitter did.”

This goes to the heart of my post. The iPod became incredibly popular because people clearly understood what it did: it served the same purpose as a portable CD/tape player, but carried the equivalent of hundreds of CD’s instead! The iPhone is still just a smartphone; it’s just got a significantly better interface. All of these technologies became popular because they did the same things their predecessors did, but better. Prediction markets haven’t become as popular as they could have been- because they do the same thing (forecasting) differently.

[Chris Masse]: “The added accuracy of the enterprise prediction markets is marginal —and anyway does not fill the gap with omniscience (contrary to people’s expectations).”

This is where I think Chris unnecessarily limits himself to examples where there is “added accuracy.” There are a LOT of applications for prediction markets where little or no forecasting is currently done; the example I commonly use is forecasting project management milestones. I think it would be ideal if a management dashboard (RAG status) was created using only the inputs of prediction markets on the probability that a project would meet its next milestones!

Sure, in cases where prediction markets are “competing” with other forecasting mechanisms, such as for demand for products down to the individual SKU level, prediction markets may not be the best tool. The power of prediction markets really comes into play in situations that are difficult to forecast without a market.

[Chris Masse]: “In the context of a Fortune-500 company, which is of course much smaller than a country, the pool of potential active participants whose trading activity is sustained over time is quite tiny.”

I just want to point out again that in my research I found that a group as small as 16 people could generate calibrated forecasts.

[Medemi]: “In order to make good predictions one needs both approaches, bottom-up as well as top-down.”

I completely agree with this. Companies can’t live on prediction markets alone, but neither should they do all of their forecasting without prediction markets!

[Medemi]: “The problem is, this valuable information (from the experts in the field) and how the problems can be solved was not passed on the management. Why? Because they are not interested. The bottom-up approach is simply NON-EXISTENT.”

I disagree slightly. I agree that certain people in an organization are dis-interested, because they’re close enough to a problem that they think they can solve it and don’t want to hear any bad news. But I have also talked to senior managers that are very interested in using prediction markets for exactly this reason; they want to know if their project managers are telling them the truth! Unfortunately, these tend to be fairly senior people in a company, and it’s tough to get in contact with them.

Summary

I hope this clarifies my position; that prediction markets are a completely different way of approaching the problem of business forecasting, and should not be pitched or considered as a replacement technology, but as a powerful but complementary technology.