Chris Masse over at Midas Oracle has recently generated a fair number comments on his post “Are IBM Smarter Cities prediction markets too smart for people?” As the commenters rightly point out, what IBM was doing was NOT a prediction market, but instead a polling system.

This is something I’ve been talking about since May 2008. When you try to use a prediction market to forecast the “best” ideas, you get a “Keynesian beauty contest.” To quote (via wikipedia):

It is not a case of choosing those [faces] that, to the best of one’s judgment, are really the prettiest, nor even those that average opinion genuinely thinks the prettiest. We have reached the third degree where we devote our intelligences to anticipating what average opinion expects the average opinion to be. (Keynes, General Theory of Employment Interest and Money, 1936).

Starting from the wrong metaphor

Ideas in an organization are a long-term investment, where significant work and development is required before you can even get to the prototype stage of development. This is a very different concept than regular, known markets, such as the number of customers in the next quarter or “will me meet sales targets next year” which are asked in prediction markets. Prediction markets work well for the regular, known markets, because prediction markets involve liquid, easily-traded contracts.

Idea markets should be seen through the metaphor of venture capital. An individual sees lots of good ideas, but none of them are well developed. Just as a venture capitalist puts resources into ideas and those resources are locked-up and illiquid until the idea has proven its success or died, an individual in an idea market should see their votes as an illiquid, sunk cost.

The organization running the idea market then concentrates their resources where individuals concentrate their votes. But just like venture capital, this cannot be a one-time market. The organization needs to regularly run similar markets, say every quarter or every six months. If an idea gets resources but doesn’t make enough progress (or doesn’t look like it will be as effective as originally thought) the market will stop voting for it. As an idea makes good progress, it continues to receive resources for development. Finally, when an idea is implemented and made successful, all of the people that voted for it during its development will be rewarded. And just like venture capital, the earlier you contribute the more you are rewarded.

The key is that idea selection and development is long-term work, and thus when markets are used to forecast and help allocate resources, the market structure must match that long-term approach.

a

From Emile of NewsFutures, here are three links and a quote from each:

The imagination market, Information Systems Frontiers (July 2007)

Participants were able to trade shares of technology ideas over the course of 3 weeks, resulting in the market identifying the “best” idea as the highest priced security. Our findings suggest that information markets for idea generation result in more ideas and more participants than traditional idea generation techniques; however, using markets to rank ideas may be no better than other methods of idea ranking.

Examining Trader Behavior in Idea Markets: An Implementation of GE’s Imagination Markets , Journal of Prediction Markets (April 2009)

In this experiment, we examine the behavior of traders that have submitted the ideas on the market and their influence on the market’s outcome. An idea’s submitter is clearly motivated to have his idea valued highly by the market, both by the funding given to the top idea as well as smaller prizes given to the top three ideas. In general, founders tended to buy their suggested ideas at prices above the volume-weighted-average-price (VWAP) in significant volumes. We discuss the implications and mitigation strategies. A survey of market participants yielded mixed results regarding the market’s effectiveness at ranking ideas but very positive results regarding the quality of ideas proposed.

GE Global Research blog (link here)

From the comments,
“GE Healthcare IT attempted an imagination market a few months ago to bring forth some new ideas for the company’s future. It left me with very strong mixed impressions: on one hand, it’s wonderful that we’re leveraging the power of technology for mass collaboration and idea sharing. On the other hand, I felt that the tool obfuscates the very opinions it seeks to gather by due to the inherent complexities of market behavior.

My primary objection is the use of the stock market paradigm to evaluate these ideas. Simply, I find it too abstract to be useful in gathering feedback about the quality of an idea. Stock investment is done by trying to predict *the change in collectively perceived value of something over time*. However, when dealing with ideas, neither the collectively perceived value, nor the change in this value over time are valuable metrics; you want people evaluating ideas based on *their opinions*, not based on their attempts to predict changes in the investment decisions of others over the course of a few weeks. These are static ideas isolated from one another, not evolving companies that interact. I think a stock market is an unnecessarily abstract, and distracting way to retrieve simple information: what do people think of these ideas?”

I’d be interested in hearing from people that disagree with this…

a

First of all, there are multiple definitions of innovation and I want to address this. Here’s a (longer) definition:

Innovation is the economically successful introduction of a new technology or new combination of existing technologies in order to create a stepwise improvement in the value (compared to the resources invested) created for the client.

The problem here is with one word: stepwise. Some people and companies think this must be a massive step; others believe that fairly small step changes can still be considered innovations. In my opinion, this difference in understanding is why the word “innovation” has become the buzzword it has.

Innovation and tools in modern corporations

I’m interested in the intersection between organisational groups and innovation. Specifically, how can new technology help?

In a previous post I discussed the various idea management software packages available to businesses and organisations to help them innovate. But what continues to bother me about all of the software packages I’ve seen is that they seem to pass on responsibility. New ideas and innovations have to go through a process of review, which raises them to increasingly higher levels of management for further review. Employees themselves don’t keep the responsibility for success. They can help forecast and vote for an idea, but that’s it.

Perhaps my military background is showing a bit, but I think much more responsibility can be pushed down to the employees themselves. When I was in the Navy, there were clear rules under which we had to operate. But within those rules, we were very free to experiment and find the best solution for our watchteam/boat/squadron. This extended down to each individual watchstation; even the most junior enlisted man on board had room in which to learn and innovate. (This doesn’t mean that the Navy is an all-innovating organisation; just that there wasn’t needless process and structure for it.)

I personally believe that each additional step of process and each additional rule limits the boundaries of innovation in an organisation. Companies must operate with rules: spending limits are musts, managers must approve formal product introductions, etc. But these are rules for the firm, not for innovation. If you start putting rules and structure around innovation, (such as each project must have a sponsor, projects must have certain approvals before they begin, etc.) a company starts down the slippery slope to irrelevance.

What matters with innovative ideas is that they get implemented. (Or at least implemented enough to “fail fast”.) Does it really matter how if they comply with the main rules in the firm? What matters is results.

Bob Sutton is a very well-known management thinker and professor at Stanford. He writes here:

innovation often happens despite rather than because of senior management, and oddly enough, the best leaders often realize that their very presence can sometimes stifle innovation.

and a fantastic story (confirmed to be true) from HP:

Some years ago, at an HP laboratory in Colorado Springs devoted to oscilloscope technology, one of our bright, energetic engineers, Chuck House, was advised to abandon a display monitor he was developing. Instead he embarked on a vacation to California —stopping along the way to show potential customers a prototype of the monitor. He wanted to find out what they thought, specifically what they wanted the product to do and what its limitations were. Their positive reaction spurred him to continue with the project, even though on his return to Colorado, he found that I, among others, had requested it be discontinued. He persuaded his R&D manager to rush the monitor into production, and as it turned out, HP sold more than 17,000 display monitors representing sales revenue of $35 million for the company.

What to do?

Fundamentally, there is a difference between coming up with the ideas and innovations and formally developing them. At what point do you make what’s really an entrepreneurial-type activity a big-company project management process? Do you do it while it’s still just an idea, or later in its life? I know where I personally come down on this question… what about you?

Future prediction markets news

I’ve been working with a media company in London to develop a public prediction market for their (industry vertical) network. It’s still softly launching, and I don’t want to steal any thunder until they’ve had a chance to fully promote it. But I look forward to discussing it in the future here.

One last note…

As I mentioned before, this blog recently achieved a Google PageRank of 6/10. Because of this I’ve been getting a LOT more spam in the comments, and have made the comments section completely moderated. But please comment below; I will approve it (hopefully) shortly thereafter.

a

What I expect for prediction markets in 2009

I’m probably going to regret doing this by the end of the year, but I’m going to put on my forecasting hat and try to predict what will happen this year in the field of prediction markets.

• Prediction markets in 2009 are going to become even more well-known and wide-spread, but there will be no single event that brings them to the attention of the public. It’s going to be a slow, but steady, growth.
• All of the prediction market vendors will mature their business offering/proposition. Both InklingMarkets and Xpree have made recent business development hires, and Inkling recently redesigned their website with a stronger business pitch and case studies. I think that business cases for prediction markets will be more clear and more developed in general because of this.
• HubDub will continue to only be the only strongly popular play-money prediction market. (HSX is probably similarly popular, but is a single-industry prediction market.) With Hubdub’s partner program and social networking features, they will see significant growth this year.
• While a couple additional software vendors may appear, I get the feeling that the market for prediction market software is largely saturated. People working on projects part-time and selling them may proliferate and be suitable for the lower end of the market, I don’t see many more serious prediction market software companies starting up in 2009.
• I’m looking forward to see how the CantorExchange develops. Will there be enough customers? Will the market see the effects of other studios trying to bid the prices of their competitors’ films down? Or will it be seen more as a financial hedging tool for institutions involved in the film industry? There are lots of open questions to evaluate here, but I still maintain it’s a great step forward for the prediction markets industry.

What to expect from this blog in 2009

I’m going to continue posting on both prediction markets, but continue talking more and more about how collective intelligence can be used in other contexts. Specifically, I’ve become really interested in how groups of people develop new ideas and new innovations.

As I’ve written before, I really don’t like the word “innovation.” Bruce Nussbaum at BusinessWeek has written a few things lately that I really like:

“Innovation” died in 2008, killed off by overuse, misuse, narrowness, incrementalism and failure to evolve. It was done in by CEOs, consultants, marketeers, advertisers and business journalists who degraded and devalued the idea by conflating it with change, technology, design, globalization, trendiness, and anything “new.” It was done it by an obsession with measurement, metrics and math and a demand for predictability in an unpredictable world.

and

“Innovation” is inadequate as a concept to deal with these changes. You have “game-changing” innovation, which is big but rare and incremental innovation which is small but common.

The truth about innovation is that it takes groups of people a lot of work to find and develop new ideas and turn them into innovations. Scott Berkun has a great chapter in his book “The Myths of Innovation” where he destroys the myth of the lone creative genius.

We know it takes groups of people to make innovation happen, so where are the tools for it? A lot of time, money and effort has gone into social networking software to connect people, such as Facebook, MySpace, etc. But I have to agree with Tim O’Reilly: we need to stop throwing sheep and do something worthy. But those tools can be very useful to us in helping foster innovation. I reviewed the software for idea and innovation software recently, but was left generally unimpressed.

These are some of my current thoughts. I plan on thinking out loud more in 2009 on these topics and seeing where it takes me. I look forward to your feedback.

a

The most popular post I’ve written to date is a review of prediction market software. Today’s post is going to be the same, but for idea/innovation software (henceforth referred to as innovation software).

Trying to even find and identify all the different types of innovation software is difficult because of the different ways people and companies think about innovation. Prediction markets are straightforward; they’re futures markets, so the software is largely the interface between the user and the order book on the database. That is not at all so for innovation software. Different people think about innovation in different ways, which I referred to in a previous post.

The list below is likely not complete, but I believe it does pick up the major players.

Digg for Ideas (ranking systems)

Salesforce – Salesforce’s solution is really well known, having been used by Starbucks in the myStarbucksIdea contest and also in Dell’s IdeaStorm. It’s a simple popularity contest, but tied in nicely with Salesforce’s platform. I’ve heard, however, that it required a significant time investment on Starbuck’s/Dell’s part in order to properly evaluate the highest-ranked ideas internally, even before they ever reached the stage of implementation.

BrightIdea – The information on BrightIdea’s software is relatively scarce; just a list of features. It looks like it’s trying to be a one-stop shop for everything, from research to idea ranking to analytics to rewards to financials and more. I understand that it’s a fairly mature product, and they have some solid clients. Overall, it’s a bit of a dark horse.

Hype Idea Management – This is a German product, and looks to be fairly basic; it’s just an idea capture and rating system. To me it looks both too basic (in general) and too complex (particularly when it comes to ranking/rating).

Idea Central from imaginatik – This is yet another piece of software that seems to exist only in a list of bulletpoints and large blocks of text. Based on their claims of paying clients it must exist and work, but I would certainly appreciate some screenshots and demos to understand what it actually focuses on.

Spigit – I’m really not sure what to think of Spigit. They look to have a fairly advanced product, which is actually three products: IdeaSpigit, InnovationSpigit, and ContestSpigit. IdeaSpigit seems to be a standard “Digg for ideas” model, where you get feedback from customers like the Salesforce IdeaExchange. InnovationSpigit is an application to use internally, with quite (and needlessly?) sophisticated algorithms to rate/rank ideas. It also bills itself as a prediction market, so I’m not sure how much of the system is a ranking/rating system and how much of it is a market-based system. Finally, ContestSpigit is the same kind of system but for a specific campaign.

Spigit seems to have a good client base and their software has won an award or two, but it’s tough to tell how useful it actually is for their clients. To me it appears to be needlessly complex, but I believe these systems should be simple and useable above all else. Their marketing positions them as a significant competitor in this industry, but I’m not sure how much is hype and how much is truth.

Market-based (aka betting-based) solutions

Nosco IdeaExchange – Nosco is a great company from Denmark that I first met a couple of years ago. They first developed a portfolio of software applications that included prediction markets and what they call an Idea Exchange. Since then they’ve found much more demand for the Idea Exchanges and have since shifted their focus to that alone.

Their Idea Exchange is still modeled off of a futures market, where you can buy and sell ideas. I still would suspect a model like this to be susceptible to gaming and in general becoming a Keynesian beauty contest. (People don’t buy what is worthwhile, they buy what they think others will think is worthwhile.) That said, they’ve what looks to be a mature product that looks fantastic and has been used by a number of Danish companies.

Consensus Point – Some clients of ConsensusPoint use their standard ForesightServer software to run prediction markets on ideas. I’ve mentioned before how prediction markets aren’t suitable for this. To ConsensusPoint’s credit, they aren’t specifically marketing a one-solution-fits-all approach; it’s just what their clients are doing with the software.

NewsFutures Idea Pageant – I really like the quote from NewsFutures on their Idea Pageant page: “A large number of ideas makes a standard prediction market approach impractical.” While I don’t think that’s the only criticism, it’s a good chunk of a start.

The Idea Pageant is a fairly straightforward and easy to understand application. Each person gets a number of positive (green) votes/tokens and a smaller number of negative (red) votes/tokens. These can be refreshed periodically, and the consistently positive ideas float to the top.

Qmarkets – Qmarkets is another prediction markets startup that seems to have also moved into the innovation software arena. There’s a fairly extensive feature list; so much so I’m not sure how to gauge exactly how complex the software actually is. Without screenshots, it’s tough to tell how well developed the solution is, but it’s certainly a potential player in the market.

Xpree – The Xpree Open Innovation Markets seems to be a bit of a hybrid solution. It has a voting system for ideas, but then later management can transfer those ideas into a prediction market. This seems to be a well-thought-out way of approaching the problem. However, I personally believe that a client could shoot themselves in the foot with poor implementation of this software. It’s all too easy to start to turn every idea into a prediction market, and that (again) is a bad plan.

Marketplace solutions (specific innovations sought)

InnoCentive – InnoCentive is one of the most well-known idea marketplaces. They’ve had some good successes so far, and substantial press coverage. I would assume they are trying to quickly ensure they take advantage of network effects and become the primary marketplace for innovators and the companies looking for innovations.

NineSigma – NineSigma is another company in this arena. It’s less a true marketplace than a forum to receive and respond to RFP’s. They do seem to have a decent client list, and have been in business since 2000.

PhilOptima – PhilOptima is a similar innovation marketplace, but is aimed at “grant makers” and thus has a different audience on the innovation seeker side.

Innovation Exchange – This site appears to be a marketplace for innovations in general, without much focus. While that’s great in principle, I think the lack of focus perhaps hurts their chances in getting significant penetration in any market, and thus any substantial market share. I mention this because there will likely be a race for market share amongst these sites, and only the winner will get the ideal network effects.

fellowforce – This is similar to Innovation Exchange, but geared more toward the Web2.0 crowd, with widgets and rankings prominently promoted.

Full-fledged Marketplace solutions (specific innovations sought and sold)

Yet2 – I’m really intrigued by this site. What’s unique is that it offers something both for companies with specific innovation needs (like the category above) but also for entrepreneurs, engineers and scientists with innovations they believe have commercial potential. While the design is a bit harsh visually, it’s a very intriguing concept.

Other

Rite-Solutions – Rite-Solutions became quite well known a couple of years ago based on a well-known article in the New York Times that discussed how they used their software to allow everyone in the company to discuss and promote their ideas internally. It’s become quite a successful product for them (though perhaps not as lucrative as some of their government/gaming industry work!).

BrainBank – BrainBank looks to be a very interesting software solution that promotes both the ranking and refining of ideas, but also some management around implementation. It’s very interesting.

MindMatters – I classified MindMatters software in this category since I couldn’t quite tell what the main purpose of the software actually is. It mentions idea capture, challenges and workflow, but it wasn’t obvious how they all fit together in their particular software application.

Summary

There are a multitude of different approaches to innovation, and there are a multitude of different software applications to help companies and organisations innovate. Are there any true market leaders? Not as far as I can tell. Some, like Salesforce, are quite well known, but aren’t necessarily that useful for a wide cross-section of companies.

This post is meant simply to review and discuss different software applications available around innovation. I plan to write much more on how innovation does and can work in organisations. You can find all of my past innovation-related posts here, and future posts will go there, too.

I sincerely look forward to your feedback.

Reminder: If you’d like to receive future posts by e-mail, just click this link to get new writings delivered directly to your inbox! And if you use an RSS Reader you can click this link to subscribe.

a

I want to begin with a pet peeve: I really don’t like the word “innovation.” I think that many people and corporations have begun turning an important concept into a buzzword.

However, innovation is still a very important concept. An organisation can grow two different ways: through market size and share, but also through innovation, generating new products and services. I firmly believe that while both are important, only innovation can keep a company healthy in the long term. But innovation happens on multiple levels. Jeff Bezos of Amazon.com said this in an interesting article in BusinessWeek:

Q: Academics say Amazon excels at different kinds of innovation — from creating new ways of doing business to making small changes that improve the online store. [...] How do you balance these approaches?

A: There is a ton of fine-grained innovation that happens on a daily basis. That kind is super important—things that make our operations more efficient and lower cost so we can afford to offer lower prices to our customers. But there is a spectrum, and at the other end is large-scale innovation like Kindle and Web services and Amazon Prime. With large-scale innovation, you have to set a very high bar. You don’t get to do too many of those [initiatives] per unit of time. You have to be really selective.

I’ve talked before about prediction markets and using them for innovation. Specifically, using PM’s to rank ideas and innovations is a horrible idea. However, I do believe that once you have a short-list of good ideas, prediction markets are appropriate to get a numerical comparison of how each could affect a business or organisation.

But so far I’ve been distinctly unimpressed with how most corporations approach “crowdsourcing” and innovation. There are two approaches, and both have a similar flaw:

• External: Companies ask their customers or the public at large to propose new ideas.
• Internal: Companies ask their employees to propose new ideas.

All of the software solutions to each of these approaches follow a similar pattern; each is essentially a Digg for ideas. But an idea is only the first step in innovation, and it’s by far the easiest step. Selecting the best ideas is only a further marginal step forward. While a “crowdsourced” method of finding and ranking ideas is great, those ideas then go through a typical corporate process to be approved, managed and implemented. What starts off so promising (getting a group of people to start the innovation process) then becomes a standard organisational process. This implementation are where great ideas go to be killed off.

Great ideas that lead to great innovations inspire passion in the people that believe in them. This passion is what is lost when an idea gets put into an standard process. While the idea still may be implemented, I think that it will generally lose momentum if the people making an idea happen weren’t those that were passionate about it.

Side note: I mentioned the difference between external and internal crowdsourcing above because there are important differences when you go beyond a Digg-style ranking mechanism. Namely, an internal approach can take advantage of great ideas without any real intellectual property concerns, where an external approach needs to be sensitive to this issue.

There are some success stories, however. When a true marketplace for ideas and innovations is established, there is a flourishing community. For example, InnoCentive is a marketplace where companies can post technical problems they’re having problems solving, and anyone can try finding the best solution. This is a great model, but only works when a company has a well-defined problem that can be addressed by the innovator. The innovators can never initiate an innovation, they can only react to a request for innovation.

Summary

Innovation is NOT about ideas. Innovation is about putting great ideas into practice. While ranking systems are appropriate and a good start to the process, they are just the easiest first step.

I am dissatisfied with the current approach to crowdsourcing and innovation because current solutions just attack a small portion of the innovation process. I have some things I’m working on that I hope will eventually bridge this gap. This is a topic I expect to write about much more in the coming months, and would appreciate any and all comments, public or private.

[End Note:] If you’d like to get new posts here delivered directly to your inbox, please click here. Or, if you read blogs via RSS, click here to subscribe. For those of you that do read this by RSS (quite a few, according to Feedburner), I recently updated the blog’s theme to make it simpler and faster loading. I hope you like it.

a

For anyone that’s read this blog, there is likely little new in the article that you haven’t heard before. It’s great that it was focused more toward corporations and how companies use them, and I hope that a steady drumbeat of articles like this will help generate the momentum for more widespread acceptance and use of prediction markets.

But I want to focus on one aspect of the article; using prediction markets for innovation. I believe that while you can use prediction markets to help quantify innovative ideas, using PM’s to prioritise ideas is a poor use of the tool.

Prediction markets exist to forecast a variable; this may be a metric (such as sales figures), or a probability (which candidate will win the election). For markets to work effectively, the variable should be well-defined, such as US sales in Q1, will the Minneapolis office open on schedule, etc. By their very nature, ideas tend to be quite poorly-defined. Before an idea is ever put into place, significant work takes place to develop it within a company.

When companies start using prediction markets to “harvest and prioritize ideas” I believe they are using a prediction market incorrectly. The traditional prediction market is run on a variable that’s both well-defined and committed. (You are going to continue selling a particular widget; the project to open the Minneapolis office is going to happen.) However, running prediction markets to assess ideas that may change before implementation or may not happen at all turns the entire process into a simple beauty pageant. A beauty pageant doesn’t find the most beautiful girl, it finds the girl that the particular judges on the day find beautiful.

The New York Times had an interesting article recently that discussed this issue from a different tack. “Is Justin Timberlake a Product of Cumulative Advantage” discussed a study which tracked how identical songs become (or didn’t become) popular in separate social groups. Similar principles easily translate to prediction markets on ideas. In a phrase, this is the problem:

social influence played as large a role in determining the market share [price] of successful songs [ideas] as differences in quality.

So what can you do to harvest collective intelligence to generate, prioritise and quantify ideas?

Go out there, generate some great ideas in your company. But don’t use a prediction market to prioritise them; social influence plays too big a role since the market is (at that point) too ill-defined and it hasn’t been committed to by the company. Prioritise them somewhere else.

Once the list of ideas has been fully developed vetted down to a handful of well-defined ideas, then a prediction market could be useful to quantify the opportunity. At this point, traders would know enough about the opportunity to judge the potential outcome of putting them into practice, and the outcome should be reasonably accurate.

To be fair to InterContinental Hotels (which is quoted as using prediction markets for this), they are actually a NewsFutures customer and use their Idea Pageant software, instead of the normal Prediction Market software. (The Idea Pageant seems to be an effective way of doing the non-prediction-market prioritising that I mentioned above.) However, there are a number of companies that do use standard prediction markets to prioritise new ideas.

If you’re using prediction markets for this, I would suggest you stop. As discussed in the New York Times article above, the first-movers and influencers are very influential when assessing new ideas. When an idea is new and ill-defined, people put much higher priority on information from their social network instead of their own impressions. If you took the same set of initial ideas to a completely different group of people in your company you may get a very different set of results. This means that you’re not judging what people in the group think collectively, you’re measuring what that particular group thinks.

a