Mercury's Blog

I’ve had a number of comments to my recent post on prediction markets and how they approach forecasting differently than other mechanisms, both here and on MidasOracle. I’d like to respond to a number of these comments here.

Comments and Criticism

[Chris Masse]: “Number one, I don’t understand why information aggregation would be a “bottom-up” approach (as opposed to “top-down”). Our traders bring bits of information to the market —but these bits of information were originally produced by the traditional sources (news, political polls, political forecasters, opinion leaders, etc.). I don’t understand why this “bottom-up” metaphor would apply to the prediction markets.”

I have a few comments for this. First, I don’t know where Chris gets the idea that “bits of information were originally produced by the traditional sources.” Corporations aren’t trading on political markets where there are polls and expert opinions. They’re trading on things that matter to their company and to their industry. While what he mentions certainly applies to public prediction markets, it’s virtually irrelevant to corporate markets. Individual traders bring their judgement and the perspective from their place in the company and their personal history, which when combined with other employees in the company is very valuable indeed.

With the “bottom-up” metaphor, I was trying to show that forecasts are built from the views and opinions of individual employees… from the bottom-up. “Top-down” is how much of traditional forecasting is done: put data through a model at corporate HQ and generate a forecast which is then distributed throughout the company… from the top-down.

[Chris Masse]: “[trying to paraphrase me] EPMs are such a novelty, and the corporate forecasters such a bunch of retarded people, that it will take decades before commercial organizations get to adopt the prediction market tool.”

It’s not that corporate forecasters are retarded, just that prediction markets are completely different to anything they’ve ever encountered for forecasting. And like anything different, they’re generally going to be ignored. Note that virtually all of the corporate prediction market trials are NOT initiated by forecasters, they’re initiated by general managers who aren’t so directly tied into a specific forecasting tool world-view.

[Chris Masse]: “If enterprise prediction markets were such a revolutionary and powerful forecasting tool, it would have found a market already —just like the iPod, the iPhone, FaceBook or Twitter did.”

This goes to the heart of my post. The iPod became incredibly popular because people clearly understood what it did: it served the same purpose as a portable CD/tape player, but carried the equivalent of hundreds of CD’s instead! The iPhone is still just a smartphone; it’s just got a significantly better interface. All of these technologies became popular because they did the same things their predecessors did, but better. Prediction markets haven’t become as popular as they could have been- because they do the same thing (forecasting) differently.

[Chris Masse]: “The added accuracy of the enterprise prediction markets is marginal —and anyway does not fill the gap with omniscience (contrary to people’s expectations).”

This is where I think Chris unnecessarily limits himself to examples where there is “added accuracy.” There are a LOT of applications for prediction markets where little or no forecasting is currently done; the example I commonly use is forecasting project management milestones. I think it would be ideal if a management dashboard (RAG status) was created using only the inputs of prediction markets on the probability that a project would meet its next milestones!

Sure, in cases where prediction markets are “competing” with other forecasting mechanisms, such as for demand for products down to the individual SKU level, prediction markets may not be the best tool. The power of prediction markets really comes into play in situations that are difficult to forecast without a market.

[Chris Masse]: “In the context of a Fortune-500 company, which is of course much smaller than a country, the pool of potential active participants whose trading activity is sustained over time is quite tiny.”

I just want to point out again that in my research I found that a group as small as 16 people could generate calibrated forecasts.

[Medemi]: “In order to make good predictions one needs both approaches, bottom-up as well as top-down.”

I completely agree with this. Companies can’t live on prediction markets alone, but neither should they do all of their forecasting without prediction markets!

[Medemi]: “The problem is, this valuable information (from the experts in the field) and how the problems can be solved was not passed on the management. Why? Because they are not interested. The bottom-up approach is simply NON-EXISTENT.”

I disagree slightly. I agree that certain people in an organization are dis-interested, because they’re close enough to a problem that they think they can solve it and don’t want to hear any bad news. But I have also talked to senior managers that are very interested in using prediction markets for exactly this reason; they want to know if their project managers are telling them the truth! Unfortunately, these tend to be fairly senior people in a company, and it’s tough to get in contact with them.

Summary

I hope this clarifies my position; that prediction markets are a completely different way of approaching the problem of business forecasting, and should not be pitched or considered as a replacement technology, but as a powerful but complementary technology.

The field of prediction markets seems to be going through a bit of a crisis of confidence recently. I would personally trace it to the recent election (where other forecasters like Nate Silver made forecasts as good as PM’s), as well as recent press like the Economist article. The general feeling is a questioning of prediction markets: if they’re so good at forecasting, why aren’t they being used much more widely? I think I have the start a good reason why.

Core Issue

Traditional forecasting is done through highly analytical techniques using past data. Statistical measures are used to generate forecasts, with probability ranges. This industry is quite large, and is highly exacting.

Prediction markets take an orthogonal approach to traditional forecasting. Instead of a “top-down” approach where huge data sets are analyzed, prediction markets use a “bottom-up” approach that combine individuals’ forecasts.

The reason prediction markets haven’t been adopted widely is because they are a tool that approaches the forecasting problem from a completely different perspective.

An example – Enterprise Business Intelligence

I’ve recently been looking into the Enterprise Business Intelligence/Business Management industry, and came across what I think is a similar phenomenon. The vast majority of the industry is composed of massive analytical solutions from the likes of SAP, Oracle, IBM, etc. They are massive companies, and implementing a “solution” can easily take a year or more. Their clients design the system from a “top-down” perspective, determining from the outset what the processes and procedures are going to be.

But then there is software like Thingamy. Thingamy is the creation of Sig Rinde, a Norwegian living in the south of France. Instead of looking at enterprise business intelligence from the top-down, he has created software that approaches the problem from the bottom-up. Instead of establishing pre-defined processes (that may not even work or will be changed by the time the software is configured), Thingamy tracks emergent processes as they happen. It can start with a very small, hard-to-define process and then scales up as the business needs it.

While Thingamy has gotten some good press and attention over the years, it’s still a fairly small company. Again, I believe this is because it takes a fundamentally different approach to the problem compared to the rest of the current industry. Hugely different approaches cause cognitive dissonance, which slow adoption.

What does this mean?

There are new types of technologies that approach business problems from entirely different directions. Prediction markets is one of these technologies. Using PM’s means companies have to upset some of their current notions about how power and influence flow in a company, relying on “soft” information from lower-level employees. A different approach also means that in certain situations they’ll be clearly superior, but also that in other situations they won’t be. Traditional methods and thumb-rules for situations just don’t automatically work.

For example, prediction markets where there is a lot of public information (like election markets) may prove to integrate new news and information more quickly, but may not be quite as accurate as other methods in the final analysis. But where information is scarce (like some internal corporate forecasts), a prediction market may be ideal. In general, new ways of thinking have to be established to know when and where to use this new tool effectively. That’s why I believe prediction markets will take quite some time to see any sort of a spike in growth; expect a slow burn for a long time.

Quick note

Just a quick note for you all. I’m curious about how well Google’s AdSense can be used to monetize a blog, so I’m going to be running AdSense on this blog on a one-month trial. If you have any opinions on this, please feel free to e-mail me or comment below.

I wanted to take a minute and mention that HubDub is running a March Madness “Predict-A-Thon” prediction market tournament. For someone like myself that hasn’t followed college basketball in a while, this is a far more interesting way to see the tournament unfold.

As I write this, Jason Trost of Smarkets fame is currently atop the leaderboard, but I expect this may very well change over the course of the next days/weeks. HubDub’s founder Nigel has climbed from the basement of the list, where he was third from the bottom in the early days!

HubDub is trying out a new model with this tournament, and will be awarding cash prizes to the top traders! If you think you’ve got what it takes just click here to sign up now.

On a final note, I’d like to say that it’s about time that my alma mater’s basketball team, the University of Michigan Wolverines, finally made it back to the NCAA tournament. Unfortunately, this was likely due to cosmic imbalance in sporting karma from the Wolverines football team collapsing this year. It would be great if they could at least make it through the first round of the tournament to start evening things out a bit.

There has recently been quite a bit of discussion on MidasOracle over the value of prediction markets. Part of this was sparked off by the recent Economist article on prediction markets, but part of it has to do with things Chris Masse has been writing for some time now. Chris is very vocal about his views, and I wanted to put a different perspective forward here.

Economist article

A recent article on prediction markets in the Economist was titled “An uncertain future.” I would describe its attitude toward prediction markets as mixed. Here are a few selected quotes:

“But although they have spread beyond early-adopting companies in the technology industry, they have still not become mainstream management tools. Even fervent advocates admit much remains to be done to convince sceptical managers of their value.”

“Koch says the results so far have been pretty accurate compared to actual outcomes, but stresses that markets are complementary to other forecasting techniques, not a substitute for them.”

“Another reason prediction markets flop is that employees cannot see how the results are used, so they lose interest. [...] its most effective trials took place in areas where managers could do something with their findings, making staff feel that trading was worthwhile.”

“Bosses may also be wary of relying on the judgments of non-experts. Yet many pilot projects run so far have shown that junior staff can often be surprisingly good forecasters.”

So I believe the article is generally positive toward the possibility of prediction markets, but generally negative toward the lack of adoption. Which is fair; though I think some good success stories have been kept quiet by some companies for competitive advantage and corporate ego.

Chris Masse’s reaction

Chris Masse has been writing about the prediction market industry for a number of years, and started his blog MidasOracle in about 2006. Strangely, Chris’ writings on prediction markets have become quite negative in recent months. As I understand it, he’s become quite irritated that prediction markets have been touted as “vastly superior” to polls in elections, and other general marketing in the industry. The front page of MidasOracle now has the following statement (his emphasis removed):

The Truth about Prediction Markets

The social utility of the prediction markets is marginal. Number one, the aggregated information has value only for the totally uninformed people (a group that comprises those who overly obsess with prediction markets and have a narrow cultural universe). Number two, the added accuracy (if any) is minute, and, anyway, doesn’t fill up the gap between expectations and omniscience (which is how people judge forecasters). In our view, the social utility of the prediction markets lays in efficiency, not in accuracy. In complicated situations, the prediction markets integrate expectations (informed by facts and expertise) much faster than the mass media do. Their accuracy/efficiency is their uniqueness. It is their velocity that we should put to work.

Remember, dear readers, you heard it here first —on Midas Oracle.

Part of my issue is his definition of “social utility”: namely that I’m not exactly sure what he’s talking about. I completely understand the concept of utility, but “social utility” is a phrase that could mean a variety of things, and I don’t know exactly what Chris means when he writes about it.

My other response to Chris is that I feel he doesn’t distinguish between public and internal prediction markets when he discusses marketing and usage of these markets. I’d like to discuss these issues here.

Public prediction markets

Public prediction markets are tricky things. They’re tricky to get a critical mass of people involved, they’re tricky to get a flow of interesting and valuable contracts, and they’re tricky when events happen that don’t easily conform to how the contracts were established. That said, there are a handful of good and interesting public PM’s. InTrade and Betfair both offer real-money markets, and the better play-money markets include Hubdub, Inkling, Newsfutures, and HSX.

These markets do get press, particularly InTrade in the run-up to the last two presidential elections. InTrade in particular seems to be irritating Chris because of the data point that they forecasted all 50 states correctly. (Which as I’ve written before probably means that the markets weren’t running as efficiently as they could; you’re supposed to get market “failures”.) And I agree with him that too much has been made of this data point.

Overall, I think Chris rightly addresses the issue of the value of public prediction markets. In many cases they’re purely entertainment, and in other cases (even the elections), they’re not necessarily better predictors than other methods. That said, they do incorporate new information faster than any other forecasting model. But I believe Chris wrongly applies these same criticisms to a completely different model, internal prediction markets.

Internal (corporate) prediction markets

Internal prediction markets seek to do either of two things:

• forecast something important to the company where they already have a prediction
• forecast something that’s never been predicted before

When it comes to the first point, forecasting something that the company already forecasts, prediction markets may or may not be an excellent solution. I’ve seen one set of markets that absolutely blew away the accuracy of current forecasts, and I’ve seen other markets that were consistent with current forecasts with little or no accuracy edge. In this case, prediction markets can serve as a fairly low-cost “reality check” on the official forecasts. When they deviate too much, it will raise a flag for investigation. This deviation is particularly helpful since prediction market forecasts are real-time, and can react instantly to new news.

I think the second point above is perhaps the most valuable opportunity. Simple ideas like using a prediction market to establish a RAG status could potentially be very powerful in a company. Depending on the value of the information forecasted, even a moderately accurate prediction market could be incredibly useful to a company.

Chris also over-reacts to some of the marketing from software vendors. Yes, some may mention events like InTrade predicting all 50 states in 2004, but that’s just a party trick to get people interested and in the door. Companies first need to be convinced that the tool works, then they need to be convinced that it works for them. The second part is done individually with their clients so we don’t see it. (Though I really like the case studies that Inkling have put on their website.)

I personally think that he gives too much credit to the popular press in raising the profile of prediction markets; far more credit is due to James Surowiecki and “The Wisdom of Crowds.” (Amazon link)

Summary

Prediction markets are interesting tools, but the lessons learned from public prediction markets are different than those learned from internal prediction markets. It’s important that these two applications are not confused. There will always (and rightly) be questions about the accuracy of prediction markets. In some cases it’s clear that markets are superior, in some internal cases it doesn’t matter since nothing is forecasted right now, and in other cases markets will be about as accurate as what’s already predicted. But in all three cases there is still a valuable argument as to why prediction markets should be used. It all comes down to the specific needs of an industry or company, which is where the vendors step in to help.

The results speak for themselves. Each year there are more software vendors, and each year the existing software vendors hire more people to serve their clients. I’m not going to try and predict the long-term future, but I believe the short-term future is positive.

Quick definition of innovation

First of all, there are multiple definitions of innovation and I want to address this. Here’s a (longer) definition:

Innovation is the economically successful introduction of a new technology or new combination of existing technologies in order to create a stepwise improvement in the value (compared to the resources invested) created for the client.

The problem here is with one word: stepwise. Some people and companies think this must be a massive step; others believe that fairly small step changes can still be considered innovations. In my opinion, this difference in understanding is why the word “innovation” has become the buzzword it has.

Innovation and tools in modern corporations

I’m interested in the intersection between organisational groups and innovation. Specifically, how can new technology help?

In a previous post I discussed the various idea management software packages available to businesses and organisations to help them innovate. But what continues to bother me about all of the software packages I’ve seen is that they seem to pass on responsibility. New ideas and innovations have to go through a process of review, which raises them to increasingly higher levels of management for further review. Employees themselves don’t keep the responsibility for success. They can help forecast and vote for an idea, but that’s it.

Perhaps my military background is showing a bit, but I think much more responsibility can be pushed down to the employees themselves. When I was in the Navy, there were clear rules under which we had to operate. But within those rules, we were very free to experiment and find the best solution for our watchteam/boat/squadron. This extended down to each individual watchstation; even the most junior enlisted man on board had room in which to learn and innovate. (This doesn’t mean that the Navy is an all-innovating organisation; just that there wasn’t needless process and structure for it.)

I personally believe that each additional step of process and each additional rule limits the boundaries of innovation in an organisation. Companies must operate with rules: spending limits are musts, managers must approve formal product introductions, etc. But these are rules for the firm, not for innovation. If you start putting rules and structure around innovation, (such as each project must have a sponsor, projects must have certain approvals before they begin, etc.) a company starts down the slippery slope to irrelevance.

What matters with innovative ideas is that they get implemented. (Or at least implemented enough to “fail fast”.) Does it really matter how if they comply with the main rules in the firm? What matters is results.

Bob Sutton is a very well-known management thinker and professor at Stanford. He writes here:

innovation often happens despite rather than because of senior management, and oddly enough, the best leaders often realize that their very presence can sometimes stifle innovation.

and a fantastic story (confirmed to be true) from HP:

Some years ago, at an HP laboratory in Colorado Springs devoted to oscilloscope technology, one of our bright, energetic engineers, Chuck House, was advised to abandon a display monitor he was developing. Instead he embarked on a vacation to California —stopping along the way to show potential customers a prototype of the monitor. He wanted to find out what they thought, specifically what they wanted the product to do and what its limitations were. Their positive reaction spurred him to continue with the project, even though on his return to Colorado, he found that I, among others, had requested it be discontinued. He persuaded his R&D manager to rush the monitor into production, and as it turned out, HP sold more than 17,000 display monitors representing sales revenue of $35 million for the company.

What to do?

Fundamentally, there is a difference between coming up with the ideas and innovations and formally developing them. At what point do you make what’s really an entrepreneurial-type activity a big-company project management process? Do you do it while it’s still just an idea, or later in its life? I know where I personally come down on this question… what about you?

Future prediction markets news

I’ve been working with a media company in London to develop a public prediction market for their (industry vertical) network. It’s still softly launching, and I don’t want to steal any thunder until they’ve had a chance to fully promote it. But I look forward to discussing it in the future here.

One last note…

As I mentioned before, this blog recently achieved a Google PageRank of 6/10. Because of this I’ve been getting a LOT more spam in the comments, and have made the comments section completely moderated. But please comment below; I will approve it (hopefully) shortly thereafter.