That's the reason we have high hopes at StyleHop that crowdsourced forecasting could make a big difference in fashion – the algorithms have never worked because of the changing trends. However, we don't use PM methodology – just too complicated for our fashionista panel.

That's the reason we have high hopes at StyleHop that crowdsourced forecasting could make a big difference in fashion – the algorithms have never worked because of the changing trends. However, we don't use PM methodology – just too complicated for our fashionista panel.

I'm copying it here since it seems relevant:

Hi, Paul.

I agree that CrowdClarity’s slides don’t get into the detail necessary to understand why they were successful. But that’s largely because those slides were published as a sales tool… a top-line attention grabber for potential sales leads.

The things I would want to understand to truly evaluate success would include: demographics of traders, the types of forecasts that already exist, what error rates are when sales volatility is more “normal”, what trader incentives were used, etc.

Regarding your point on number of traders, my own research showed that as long as you have more than 15 traders, the market generates a calibrated result. (The markets I ran were probabilistic.) So that’s actually quite a realistic number, even though it might intuitively seem low. But I would point out that because of the nature of the markets I ran, the likelihood that traders knew each other was low, so there was a natural diversity in the 15+ that I studied.

I would certainly consider the prediction market / case study CrowdClarity published a “success.” The key for me is turning that into a *business* success. You’re very right that if this was one of the first months that a PM was run, no one would have likely believed the results, even if they were the closest to the eventual truth.

But let’s say that a PM has been showing reasonable accuracy for several months, which I would define as similar or better accuracy than other forecasting methods. Then, like the case study, the prediction market shows a drastically different result than any other forecast. While a manager probably won’t “bet the farm” on a prediction market alone, that certainly would warrant re-thinking the forecast.

The reason is simply that the market aggregates the information more quickly than other methods. I would argue that GM and Edmunds.com used forecasting models that are quite good most of the time, but completely wrong when some of the basic assumptions collapse. Since the PM doesn’t rely on algorithms, collapsing assumptions won’t affect accuracy.

In many ways, I think this last point is the most important with prediction markets. 80-90% of the time a prediction market might generate a forecast with accuracy that’s on par with other methods… maybe a little better, maybe a little worse. But the other 10-20% of the time, when the forecasts diverge significantly, is where prediction markets can be *very* useful. When assumptions behind traditional models weaken or collapse, a prediction market can be the early warning signal, since it uses a different methodology to generate a forecast.

Maybe a company spends $3-4k a year on a prediction market that generally confirms or better brackets existing forecasts 11 months out of the year. But if the intelligence it generates that 12th month of the year helps the company save $20k, it strikes me as a wise investment.

Again, that’s not to say that a prediction market is going to solve a company’s problems. It needs to address problems where the cost of the error is worth the investment and where a prediction market can effectively address it. And that’s a sensitive balance.

Thanks, good post. Here is a another new (to me at least) player.

http://wiscom.co.il/

-j

I echo both your sentiments.

Very good fortunes to all at CrowdClarity.

John Delaney
CEO
Intrade

~alex