Mercury's Blog

I’ve had a number of comments to my recent post on prediction markets and how they approach forecasting differently than other mechanisms, both here and on MidasOracle. I’d like to respond to a number of these comments here.

Comments and Criticism

[Chris Masse]: “Number one, I don’t understand why information aggregation would be a “bottom-up” approach (as opposed to “top-down”). Our traders bring bits of information to the market —but these bits of information were originally produced by the traditional sources (news, political polls, political forecasters, opinion leaders, etc.). I don’t understand why this “bottom-up” metaphor would apply to the prediction markets.”

I have a few comments for this. First, I don’t know where Chris gets the idea that “bits of information were originally produced by the traditional sources.” Corporations aren’t trading on political markets where there are polls and expert opinions. They’re trading on things that matter to their company and to their industry. While what he mentions certainly applies to public prediction markets, it’s virtually irrelevant to corporate markets. Individual traders bring their judgement and the perspective from their place in the company and their personal history, which when combined with other employees in the company is very valuable indeed.

With the “bottom-up” metaphor, I was trying to show that forecasts are built from the views and opinions of individual employees… from the bottom-up. “Top-down” is how much of traditional forecasting is done: put data through a model at corporate HQ and generate a forecast which is then distributed throughout the company… from the top-down.

[Chris Masse]: “[trying to paraphrase me] EPMs are such a novelty, and the corporate forecasters such a bunch of retarded people, that it will take decades before commercial organizations get to adopt the prediction market tool.”

It’s not that corporate forecasters are retarded, just that prediction markets are completely different to anything they’ve ever encountered for forecasting. And like anything different, they’re generally going to be ignored. Note that virtually all of the corporate prediction market trials are NOT initiated by forecasters, they’re initiated by general managers who aren’t so directly tied into a specific forecasting tool world-view.

[Chris Masse]: “If enterprise prediction markets were such a revolutionary and powerful forecasting tool, it would have found a market already —just like the iPod, the iPhone, FaceBook or Twitter did.”

This goes to the heart of my post. The iPod became incredibly popular because people clearly understood what it did: it served the same purpose as a portable CD/tape player, but carried the equivalent of hundreds of CD’s instead! The iPhone is still just a smartphone; it’s just got a significantly better interface. All of these technologies became popular because they did the same things their predecessors did, but better. Prediction markets haven’t become as popular as they could have been- because they do the same thing (forecasting) differently.

[Chris Masse]: “The added accuracy of the enterprise prediction markets is marginal —and anyway does not fill the gap with omniscience (contrary to people’s expectations).”

This is where I think Chris unnecessarily limits himself to examples where there is “added accuracy.” There are a LOT of applications for prediction markets where little or no forecasting is currently done; the example I commonly use is forecasting project management milestones. I think it would be ideal if a management dashboard (RAG status) was created using only the inputs of prediction markets on the probability that a project would meet its next milestones!

Sure, in cases where prediction markets are “competing” with other forecasting mechanisms, such as for demand for products down to the individual SKU level, prediction markets may not be the best tool. The power of prediction markets really comes into play in situations that are difficult to forecast without a market.

[Chris Masse]: “In the context of a Fortune-500 company, which is of course much smaller than a country, the pool of potential active participants whose trading activity is sustained over time is quite tiny.”

I just want to point out again that in my research I found that a group as small as 16 people could generate calibrated forecasts.

[Medemi]: “In order to make good predictions one needs both approaches, bottom-up as well as top-down.”

I completely agree with this. Companies can’t live on prediction markets alone, but neither should they do all of their forecasting without prediction markets!

[Medemi]: “The problem is, this valuable information (from the experts in the field) and how the problems can be solved was not passed on the management. Why? Because they are not interested. The bottom-up approach is simply NON-EXISTENT.”

I disagree slightly. I agree that certain people in an organization are dis-interested, because they’re close enough to a problem that they think they can solve it and don’t want to hear any bad news. But I have also talked to senior managers that are very interested in using prediction markets for exactly this reason; they want to know if their project managers are telling them the truth! Unfortunately, these tend to be fairly senior people in a company, and it’s tough to get in contact with them.

Summary

I hope this clarifies my position; that prediction markets are a completely different way of approaching the problem of business forecasting, and should not be pitched or considered as a replacement technology, but as a powerful but complementary technology.

The field of prediction markets seems to be going through a bit of a crisis of confidence recently. I would personally trace it to the recent election (where other forecasters like Nate Silver made forecasts as good as PM’s), as well as recent press like the Economist article. The general feeling is a questioning of prediction markets: if they’re so good at forecasting, why aren’t they being used much more widely? I think I have the start a good reason why.

Core Issue

Traditional forecasting is done through highly analytical techniques using past data. Statistical measures are used to generate forecasts, with probability ranges. This industry is quite large, and is highly exacting.

Prediction markets take an orthogonal approach to traditional forecasting. Instead of a “top-down” approach where huge data sets are analyzed, prediction markets use a “bottom-up” approach that combine individuals’ forecasts.

The reason prediction markets haven’t been adopted widely is because they are a tool that approaches the forecasting problem from a completely different perspective.

An example – Enterprise Business Intelligence

I’ve recently been looking into the Enterprise Business Intelligence/Business Management industry, and came across what I think is a similar phenomenon. The vast majority of the industry is composed of massive analytical solutions from the likes of SAP, Oracle, IBM, etc. They are massive companies, and implementing a “solution” can easily take a year or more. Their clients design the system from a “top-down” perspective, determining from the outset what the processes and procedures are going to be.

But then there is software like Thingamy. Thingamy is the creation of Sig Rinde, a Norwegian living in the south of France. Instead of looking at enterprise business intelligence from the top-down, he has created software that approaches the problem from the bottom-up. Instead of establishing pre-defined processes (that may not even work or will be changed by the time the software is configured), Thingamy tracks emergent processes as they happen. It can start with a very small, hard-to-define process and then scales up as the business needs it.

While Thingamy has gotten some good press and attention over the years, it’s still a fairly small company. Again, I believe this is because it takes a fundamentally different approach to the problem compared to the rest of the current industry. Hugely different approaches cause cognitive dissonance, which slow adoption.

What does this mean?

There are new types of technologies that approach business problems from entirely different directions. Prediction markets is one of these technologies. Using PM’s means companies have to upset some of their current notions about how power and influence flow in a company, relying on “soft” information from lower-level employees. A different approach also means that in certain situations they’ll be clearly superior, but also that in other situations they won’t be. Traditional methods and thumb-rules for situations just don’t automatically work.

For example, prediction markets where there is a lot of public information (like election markets) may prove to integrate new news and information more quickly, but may not be quite as accurate as other methods in the final analysis. But where information is scarce (like some internal corporate forecasts), a prediction market may be ideal. In general, new ways of thinking have to be established to know when and where to use this new tool effectively. That’s why I believe prediction markets will take quite some time to see any sort of a spike in growth; expect a slow burn for a long time.

Quick note

Just a quick note for you all. I’m curious about how well Google’s AdSense can be used to monetize a blog, so I’m going to be running AdSense on this blog on a one-month trial. If you have any opinions on this, please feel free to e-mail me or comment below.

I wanted to take a minute and mention that HubDub is running a March Madness “Predict-A-Thon” prediction market tournament. For someone like myself that hasn’t followed college basketball in a while, this is a far more interesting way to see the tournament unfold.

As I write this, Jason Trost of Smarkets fame is currently atop the leaderboard, but I expect this may very well change over the course of the next days/weeks. HubDub’s founder Nigel has climbed from the basement of the list, where he was third from the bottom in the early days!

HubDub is trying out a new model with this tournament, and will be awarding cash prizes to the top traders! If you think you’ve got what it takes just click here to sign up now.

On a final note, I’d like to say that it’s about time that my alma mater’s basketball team, the University of Michigan Wolverines, finally made it back to the NCAA tournament. Unfortunately, this was likely due to cosmic imbalance in sporting karma from the Wolverines football team collapsing this year. It would be great if they could at least make it through the first round of the tournament to start evening things out a bit.

There has recently been quite a bit of discussion on MidasOracle over the value of prediction markets. Part of this was sparked off by the recent Economist article on prediction markets, but part of it has to do with things Chris Masse has been writing for some time now. Chris is very vocal about his views, and I wanted to put a different perspective forward here.

Economist article

A recent article on prediction markets in the Economist was titled “An uncertain future.” I would describe its attitude toward prediction markets as mixed. Here are a few selected quotes:

“But although they have spread beyond early-adopting companies in the technology industry, they have still not become mainstream management tools. Even fervent advocates admit much remains to be done to convince sceptical managers of their value.”

“Koch says the results so far have been pretty accurate compared to actual outcomes, but stresses that markets are complementary to other forecasting techniques, not a substitute for them.”

“Another reason prediction markets flop is that employees cannot see how the results are used, so they lose interest. [...] its most effective trials took place in areas where managers could do something with their findings, making staff feel that trading was worthwhile.”

“Bosses may also be wary of relying on the judgments of non-experts. Yet many pilot projects run so far have shown that junior staff can often be surprisingly good forecasters.”

So I believe the article is generally positive toward the possibility of prediction markets, but generally negative toward the lack of adoption. Which is fair; though I think some good success stories have been kept quiet by some companies for competitive advantage and corporate ego.

Chris Masse’s reaction

Chris Masse has been writing about the prediction market industry for a number of years, and started his blog MidasOracle in about 2006. Strangely, Chris’ writings on prediction markets have become quite negative in recent months. As I understand it, he’s become quite irritated that prediction markets have been touted as “vastly superior” to polls in elections, and other general marketing in the industry. The front page of MidasOracle now has the following statement (his emphasis removed):

The Truth about Prediction Markets

The social utility of the prediction markets is marginal. Number one, the aggregated information has value only for the totally uninformed people (a group that comprises those who overly obsess with prediction markets and have a narrow cultural universe). Number two, the added accuracy (if any) is minute, and, anyway, doesn’t fill up the gap between expectations and omniscience (which is how people judge forecasters). In our view, the social utility of the prediction markets lays in efficiency, not in accuracy. In complicated situations, the prediction markets integrate expectations (informed by facts and expertise) much faster than the mass media do. Their accuracy/efficiency is their uniqueness. It is their velocity that we should put to work.

Remember, dear readers, you heard it here first —on Midas Oracle.

Part of my issue is his definition of “social utility”: namely that I’m not exactly sure what he’s talking about. I completely understand the concept of utility, but “social utility” is a phrase that could mean a variety of things, and I don’t know exactly what Chris means when he writes about it.

My other response to Chris is that I feel he doesn’t distinguish between public and internal prediction markets when he discusses marketing and usage of these markets. I’d like to discuss these issues here.

Public prediction markets

Public prediction markets are tricky things. They’re tricky to get a critical mass of people involved, they’re tricky to get a flow of interesting and valuable contracts, and they’re tricky when events happen that don’t easily conform to how the contracts were established. That said, there are a handful of good and interesting public PM’s. InTrade and Betfair both offer real-money markets, and the better play-money markets include Hubdub, Inkling, Newsfutures, and HSX.

These markets do get press, particularly InTrade in the run-up to the last two presidential elections. InTrade in particular seems to be irritating Chris because of the data point that they forecasted all 50 states correctly. (Which as I’ve written before probably means that the markets weren’t running as efficiently as they could; you’re supposed to get market “failures”.) And I agree with him that too much has been made of this data point.

Overall, I think Chris rightly addresses the issue of the value of public prediction markets. In many cases they’re purely entertainment, and in other cases (even the elections), they’re not necessarily better predictors than other methods. That said, they do incorporate new information faster than any other forecasting model. But I believe Chris wrongly applies these same criticisms to a completely different model, internal prediction markets.

Internal (corporate) prediction markets

Internal prediction markets seek to do either of two things:

• forecast something important to the company where they already have a prediction
• forecast something that’s never been predicted before

When it comes to the first point, forecasting something that the company already forecasts, prediction markets may or may not be an excellent solution. I’ve seen one set of markets that absolutely blew away the accuracy of current forecasts, and I’ve seen other markets that were consistent with current forecasts with little or no accuracy edge. In this case, prediction markets can serve as a fairly low-cost “reality check” on the official forecasts. When they deviate too much, it will raise a flag for investigation. This deviation is particularly helpful since prediction market forecasts are real-time, and can react instantly to new news.

I think the second point above is perhaps the most valuable opportunity. Simple ideas like using a prediction market to establish a RAG status could potentially be very powerful in a company. Depending on the value of the information forecasted, even a moderately accurate prediction market could be incredibly useful to a company.

Chris also over-reacts to some of the marketing from software vendors. Yes, some may mention events like InTrade predicting all 50 states in 2004, but that’s just a party trick to get people interested and in the door. Companies first need to be convinced that the tool works, then they need to be convinced that it works for them. The second part is done individually with their clients so we don’t see it. (Though I really like the case studies that Inkling have put on their website.)

I personally think that he gives too much credit to the popular press in raising the profile of prediction markets; far more credit is due to James Surowiecki and “The Wisdom of Crowds.” (Amazon link)

Summary

Prediction markets are interesting tools, but the lessons learned from public prediction markets are different than those learned from internal prediction markets. It’s important that these two applications are not confused. There will always (and rightly) be questions about the accuracy of prediction markets. In some cases it’s clear that markets are superior, in some internal cases it doesn’t matter since nothing is forecasted right now, and in other cases markets will be about as accurate as what’s already predicted. But in all three cases there is still a valuable argument as to why prediction markets should be used. It all comes down to the specific needs of an industry or company, which is where the vendors step in to help.

The results speak for themselves. Each year there are more software vendors, and each year the existing software vendors hire more people to serve their clients. I’m not going to try and predict the long-term future, but I believe the short-term future is positive.

Happy New Year to everyone! I hope that 2009 is happy, healthy and prosperous for you. I specifically wanted to thank you as a reader of this blog, and those who have linked to my posts here. Google recently refreshed their PageRank for Mercury’s Blog, and it now has a PageRank of 6/10! Though I don’t post often, I do try to write long-form news and analysis, and I’m glad you’ve found it useful.

What I expect for prediction markets in 2009

I’m probably going to regret doing this by the end of the year, but I’m going to put on my forecasting hat and try to predict what will happen this year in the field of prediction markets.

• Prediction markets in 2009 are going to become even more well-known and wide-spread, but there will be no single event that brings them to the attention of the public. It’s going to be a slow, but steady, growth.
• All of the prediction market vendors will mature their business offering/proposition. Both InklingMarkets and Xpree have made recent business development hires, and Inkling recently redesigned their website with a stronger business pitch and case studies. I think that business cases for prediction markets will be more clear and more developed in general because of this.
• HubDub will continue to only be the only strongly popular play-money prediction market. (HSX is probably similarly popular, but is a single-industry prediction market.) With Hubdub’s partner program and social networking features, they will see significant growth this year.
• While a couple additional software vendors may appear, I get the feeling that the market for prediction market software is largely saturated. People working on projects part-time and selling them may proliferate and be suitable for the lower end of the market, I don’t see many more serious prediction market software companies starting up in 2009.
• I’m looking forward to see how the CantorExchange develops. Will there be enough customers? Will the market see the effects of other studios trying to bid the prices of their competitors’ films down? Or will it be seen more as a financial hedging tool for institutions involved in the film industry? There are lots of open questions to evaluate here, but I still maintain it’s a great step forward for the prediction markets industry.

What to expect from this blog in 2009

I’m going to continue posting on both prediction markets, but continue talking more and more about how collective intelligence can be used in other contexts. Specifically, I’ve become really interested in how groups of people develop new ideas and new innovations.

As I’ve written before, I really don’t like the word “innovation.” Bruce Nussbaum at BusinessWeek has written a few things lately that I really like:

“Innovation” died in 2008, killed off by overuse, misuse, narrowness, incrementalism and failure to evolve. It was done in by CEOs, consultants, marketeers, advertisers and business journalists who degraded and devalued the idea by conflating it with change, technology, design, globalization, trendiness, and anything “new.” It was done it by an obsession with measurement, metrics and math and a demand for predictability in an unpredictable world.

and

“Innovation” is inadequate as a concept to deal with these changes. You have “game-changing” innovation, which is big but rare and incremental innovation which is small but common.

The truth about innovation is that it takes groups of people a lot of work to find and develop new ideas and turn them into innovations. Scott Berkun has a great chapter in his book “The Myths of Innovation” where he destroys the myth of the lone creative genius.

We know it takes groups of people to make innovation happen, so where are the tools for it? A lot of time, money and effort has gone into social networking software to connect people, such as Facebook, MySpace, etc. But I have to agree with Tim O’Reilly: we need to stop throwing sheep and do something worthy. But those tools can be very useful to us in helping foster innovation. I reviewed the software for idea and innovation software recently, but was left generally unimpressed.

These are some of my current thoughts. I plan on thinking out loud more in 2009 on these topics and seeing where it takes me. I look forward to your feedback.