A new competitor in prediction markets, and their brilliant case study
September 28th, 2009I recently found out about a new competitor in corporate prediction markets: CrowdClarity.
I’m a little partial to these guys, mainly because they come from my alma mater, the University of Michigan. The key people look to be a mix of entrepreneurial students and professionals. The company itself was started a year and a half ago, but has had success with early pilot projects.
In fact, three slides included in their “Learn more” online slideshow are quite powerful statements as to why prediction markets can be useful. They were predicting car sales in the winter of 2008/2009, during what was one of the most volatile months the industry has seen in many, many years. And the prediction market beat the internal forecast made at the beginning of the month, and the expert forecast made at the end of the month.
To recap, the prediction market beat the official GM forecast (made at the beginning of the month) easily, which isn’t hugely surprising considering the myopic nature of internal forecasting. But the prediction market also beat the Edmunds.com forecast. This is particularly interesting, as Edmunds would have had the opportunity to review almost the entire month’s news and data before making their forecast at the end of the month.
Examining the numbers
Let’s quickly quantify this error. Assume an average Chevrolet sells for $18,000. After dealer markup, assume that GM/Chevrolet receives $16,000 per vehicle.
Within the first week of November 2008, the prediction market would have warned Chevrolet that they were going to miss their revenue targets by $800 million in the Chevrolet division alone. And depending upon the exact product mix, this could have easily exceeded $1billion.
Now Chris can blather on about corporate prediction markets, but he’s simply wrong. Assume that even with three weeks’ early warning Chevrolet was only able to save 10% of that gap, it’s still $80million in savings. Even if a corporate prediction market for a giant company like GM cost $200,000 a year, that would still be a return on investment of 40,000 %. And again, that’s in the Chevrolet division alone. (Note: It would be a rare prediction market that cost $200k/year to run.)
Now not every problem should be solved by a prediction market. This is where management expertise comes in: are the errors large enough to warrant the cost of reducing those errors? But big problems with big numbers are often very suitable to address with a prediction market.
Summary
I’d like to wish good luck to the CrowdClarity team. It’s great to see Wolverine entrepreneurs working on prediction markets. There are more and more players each year in the corporate prediction market scene, but with case studies like this behind their belts, they’ll be well-placed to pick up some business.
September 28th, 2009 at 5:48 am
Jed, good find. Nice to hear about a new and worthy prediction market startup, and a positive corporate PM case study.
~alex
September 28th, 2009 at 6:39 am
Jed and Alex,
I echo both your sentiments.
Very good fortunes to all at CrowdClarity.
John Delaney
CEO
Intrade
September 28th, 2009 at 12:30 pm
[...] Finally, a positive corporate prediction market case study… —well, according to Jed Christiansen Written by Chris F. Masse on September 28, 2009 — Leave a Comment Jed Christiansen: [...]
September 28th, 2009 at 9:40 am
Thanks for sharing, Jed. Great Slides by the CrowdClarity team.
September 28th, 2009 at 3:03 pm
Hi Jed –
Thanks, good post. Here is a another new (to me at least) player.
http://wiscom.co.il/
-j
September 29th, 2009 at 5:56 am
[...] Paul Hewitt’s analysis is more interesting than Jed Christiansen’s naive take. [...]
September 29th, 2009 at 6:11 am
[...] Look at Jed’s post and the 4 comments below his post: “CrowdClarity is magic, and prediction markets are magic.“ — {Surprise, surprise: All the people but one are selling prediction market solutions. [...]
September 29th, 2009 at 10:36 am
I just made a substantial comment over on Midas Oracle here: http://www.midasoracle.org/2009/09/28/finally-a...
I'm copying it here since it seems relevant:
Hi, Paul.
I agree that CrowdClarity’s slides don’t get into the detail necessary to understand why they were successful. But that’s largely because those slides were published as a sales tool… a top-line attention grabber for potential sales leads.
The things I would want to understand to truly evaluate success would include: demographics of traders, the types of forecasts that already exist, what error rates are when sales volatility is more “normal”, what trader incentives were used, etc.
Regarding your point on number of traders, my own research showed that as long as you have more than 15 traders, the market generates a calibrated result. (The markets I ran were probabilistic.) So that’s actually quite a realistic number, even though it might intuitively seem low. But I would point out that because of the nature of the markets I ran, the likelihood that traders knew each other was low, so there was a natural diversity in the 15+ that I studied.
I would certainly consider the prediction market / case study CrowdClarity published a “success.” The key for me is turning that into a *business* success. You’re very right that if this was one of the first months that a PM was run, no one would have likely believed the results, even if they were the closest to the eventual truth.
But let’s say that a PM has been showing reasonable accuracy for several months, which I would define as similar or better accuracy than other forecasting methods. Then, like the case study, the prediction market shows a drastically different result than any other forecast. While a manager probably won’t “bet the farm” on a prediction market alone, that certainly would warrant re-thinking the forecast.
The reason is simply that the market aggregates the information more quickly than other methods. I would argue that GM and Edmunds.com used forecasting models that are quite good most of the time, but completely wrong when some of the basic assumptions collapse. Since the PM doesn’t rely on algorithms, collapsing assumptions won’t affect accuracy.
In many ways, I think this last point is the most important with prediction markets. 80-90% of the time a prediction market might generate a forecast with accuracy that’s on par with other methods… maybe a little better, maybe a little worse. But the other 10-20% of the time, when the forecasts diverge significantly, is where prediction markets can be *very* useful. When assumptions behind traditional models weaken or collapse, a prediction market can be the early warning signal, since it uses a different methodology to generate a forecast.
Maybe a company spends $3-4k a year on a prediction market that generally confirms or better brackets existing forecasts 11 months out of the year. But if the intelligence it generates that 12th month of the year helps the company save $20k, it strikes me as a wise investment.
Again, that’s not to say that a prediction market is going to solve a company’s problems. It needs to address problems where the cost of the error is worth the investment and where a prediction market can effectively address it. And that’s a sensitive balance.
October 1st, 2009 at 11:42 am
Jed – great comment. Really appreciate your talking about how prediction market gets around problem of algorithms not responding quickly enough to dramatic market changes.
That's the reason we have high hopes at StyleHop that crowdsourced forecasting could make a big difference in fashion – the algorithms have never worked because of the changing trends. However, we don't use PM methodology – just too complicated for our fashionista panel.
October 1st, 2009 at 5:42 pm
Jed – great comment. Really appreciate your talking about how prediction market gets around problem of algorithms not responding quickly enough to dramatic market changes.
That's the reason we have high hopes at StyleHop that crowdsourced forecasting could make a big difference in fashion – the algorithms have never worked because of the changing trends. However, we don't use PM methodology – just too complicated for our fashionista panel.