Mercury's Blog

There is a huge mis-conception in the media when it comes to evaluating the success of prediction markets in the recent US election. Simply put, you have to be wrong to be right. But depending on the type of prediction market you are operating, different methods of assessment are required. I’m going to go through each type of market and compare it to the best-of-class of poll aggregators (fivethirtyeight.com), discuss what they came up with, and potentially find a potential winner from the 2008 forecasts.

Probabilistic predictions (and prediction markets)

The most popular prediction markets this election season were probabilistic markets, where the payoff was either 1 or 0. (In the case of InTrade, $10 or $0.) The market price can and is interpreted as a percent probability that the contract will take place; one of the candidates getting elected. Unfortunately, many commentators believe that once a contract is above 50% that the candidate will win, and if a candidate is above 50% in a given market and doesn’t win, the prediction market has failed.

This is completely wrong.

I’ve discussed in previous posts that you have to be wrong in order to be right. As a quick reminder:

• 98% chance = 1 in 50 will be wrong
• 90% chance = 1 in 10 will be wrong
• 80% chance = 1 in 5 will be wrong
• 75% chance = 1 in 4 will be wrong
• 67% chance = 1 in 3 will be wrong
• 50% chance = 1 in 2 will be wrong

So how did the prediction markets do? And how did fivethirtyeight.com do in comparison? (They’re the only site I know of that takes poll results and turns them into probabilistic forecasts mathematically.) Here’s a quick snapshot:

InTrade results

We immediately run into the primary problem with evaluating these markets… while there are 50+ markets, only a small number of these were not at the extremes of the scales. I’ve just included those with a market price of <90%.

This looks great for InTrade, but it's deceiving. For example, let's take a look at all contracts at approximately 80%. (FL, OH, VA, NV, and CO). They have an average market price of 82.6%. If InTrade was perfectly calibrated, one of these should have been wrong.

It looks better on the other side. If you calculate MT, GA, ND, and IN they have an average market price of 26.75%. In fact, one of these four were wrong (Indiana).

For the mathematical amongst you, it should be clear that this type of assessment is quite crude. When you have so few data points, you need to pick and choose "bins" of data with your own best judgement. Ideally there would be enough markets that you could create strict "bins" of data and measure against those. (This is exactly how Inkling created their plots here… I would encourage you to read their post, too.) Unfortunately, we only have a small number of “battleground” markets and this just isn’t possible.

Fivethirtyeight.com results

In my opinion, fivethirtyeight.com appears to have done marginally better. Take the bottom five markets shown, which have a combined probability of 15%. In fact, one of these five actually occurred; a 20% success rate.

Another slice of data shows the same thing: MT, ND, IN, and MO have an average probability of 30% and an actual success rate of 25%. Taking MO, NC and FL the probability was 61%, and and actual success rate of 67%.

I say that fivethirtyeight.com has done marginally better because I could take reasonable slices of data throughout their predictions and come up with reasonably calibrated results. I had to specifically pick and choose to find similar results with InTrade. In other words, the fivethirtyeight.com forecasts were more internally consistent.

Non-probability predictions (and prediction markets)

Some of the lesser-cited and more lightly-traded prediction markets for the 2008 election cycle were on index markets on the vote share for each candidate. Here was the final tally of the national vote:

• Democrat – Obama – 52.7%
• Republican – McCain – 46.0%
• Other – 1.3%

What did the Iowa Electronic Market forecast? (Data from midnight on the 3rd)

• Democrat – Obama – 53.5% (0.8% error)
• Republican – McCain – 46.4% (0.4% error)

What did fivethirtyeight.com forecast?

• Democrat – Obama – 52.3% (0.4% error)
• Republican – McCain – 46.2% (0.2% error)
• Other – 1.5% (0.2% error)

To be fair, fivethirtyeight.com was the best of the “poll aggregators”. Real Clear Politics and Pollster.com came up with the following:

• Democrat – Obama – 52.1%, 52.0%
• Republican – McCain – 44.5%, 44.4%

It’s clear from this data that while the Iowa Electronic Markets were quite accurate, fivethirtyeight.com forecasts were even more accurate.

For reference, redbluerichpoor.com showed the following result from fivethirtyeight.com’s final forecasts (of state vote share) which were pretty accurate:

The issue of time

Recently, George Neumann of the Iowa Electronic Markets sent out a document to the Prediction Markets Google group that claimed that the IEM “continue to dominate polls.” One specific line really struck me as ridiculous:

During this 886 day period the average absolute error was 1.2%, amazingly similar to the final polling results but for a much longer period.

So now we’re supposed to assess the accuracy of a prediction market over a 2+ year period?!? So as the race moves and swings, and the markets with it, the IEM seems only to be concerned about the average. This implies that the election is much more static than I believe it to be.

Face it, the race changes. If the election was held within a week of the Republican convention, John McCain would have likely won… the markets reflected that. But events change, and the market changes along with them. This is why I disagree with Peter McClusky’s analysis of price changes here.

Uncertainty is priced into prediction market prices; in the fall Nate Silver of fivethirtyeight.com was consistently predicting a much higher probability of Obama winning the race than prediction markets, because the uncertainty of events between the forecast date and the election was priced into the market. As the election date got closer the uncertainty was removed from the price and Obama’s price went up. But this happens rather late in election futures; in election after election we’ve seen examples of late-breaking news that has had the ability to shift the outcome of a race.

Where does this leave prediction markets?

Were prediction markets consistently better than a well-performing site like fivethirtyeight.com? No.

Were prediction markets consistently worse than a site like fivethirtyeight.com? No.

They perform largely the same, though the final accuracy of fivethirtyeight.com was a tad bit better than InTrade/IEM. But the purpose of the two types of sites are different.

Fivethirtyeight.com and similar sites take current data and process it to extrapolate trends. These sites lag real events; Nate Silver mentioned on a number of times that he expected his model’s forecast to move, but that it hadn’t because the relevant polls hadn’t hit the model yet. Between the time a poll closes, the result released and then incorporated into the model is anything from a day or two to several days. So while it looks quite accurate, a poll aggregator is a lagging indicator. (Another couple of election cycles will tell us if their accuracy continues or was just a fluke in 2008.)

Prediction markets show the results of what a group of traders believe what will happen. This includes polling data, but also reacts to real-time information. A candidate makes a huge gaffe, and the market price will reflect it in minutes, where a poll aggregator could take days to see any effect.

This is the “social utility” of prediction markets, to answer a question that Chris Masse always poses. While they are on par with the accuracy of the best poll aggregators, their forecasts are real-time and reflect the state of the race right now. No other mechanism does this. While markets are certainly fed by polls, that isn’t the whole puzzle in and of itself.

Prediction markets worked quite well again this election cycle. Though their final forecasts were on par with the best poll aggregators, their real-time forecasts throughout the election season is the reason why they should be examined and discussed more broadly.

The most popular post I’ve written to date is a review of prediction market software. Today’s post is going to be the same, but for idea/innovation software (henceforth referred to as innovation software).

Trying to even find and identify all the different types of innovation software is difficult because of the different ways people and companies think about innovation. Prediction markets are straightforward; they’re futures markets, so the software is largely the interface between the user and the order book on the database. That is not at all so for innovation software. Different people think about innovation in different ways, which I referred to in a previous post.

The list below is likely not complete, but I believe it does pick up the major players.

Digg for Ideas (ranking systems)

Salesforce – Salesforce’s solution is really well known, having been used by Starbucks in the myStarbucksIdea contest and also in Dell’s IdeaStorm. It’s a simple popularity contest, but tied in nicely with Salesforce’s platform. I’ve heard, however, that it required a significant time investment on Starbuck’s/Dell’s part in order to properly evaluate the highest-ranked ideas internally, even before they ever reached the stage of implementation.

BrightIdea – The information on BrightIdea’s software is relatively scarce; just a list of features. It looks like it’s trying to be a one-stop shop for everything, from research to idea ranking to analytics to rewards to financials and more. I understand that it’s a fairly mature product, and they have some solid clients. Overall, it’s a bit of a dark horse.

Hype Idea Management – This is a German product, and looks to be fairly basic; it’s just an idea capture and rating system. To me it looks both too basic (in general) and too complex (particularly when it comes to ranking/rating).

Idea Central from imaginatik – This is yet another piece of software that seems to exist only in a list of bulletpoints and large blocks of text. Based on their claims of paying clients it must exist and work, but I would certainly appreciate some screenshots and demos to understand what it actually focuses on.

Spigit – I’m really not sure what to think of Spigit. They look to have a fairly advanced product, which is actually three products: IdeaSpigit, InnovationSpigit, and ContestSpigit. IdeaSpigit seems to be a standard “Digg for ideas” model, where you get feedback from customers like the Salesforce IdeaExchange. InnovationSpigit is an application to use internally, with quite (and needlessly?) sophisticated algorithms to rate/rank ideas. It also bills itself as a prediction market, so I’m not sure how much of the system is a ranking/rating system and how much of it is a market-based system. Finally, ContestSpigit is the same kind of system but for a specific campaign.

Spigit seems to have a good client base and their software has won an award or two, but it’s tough to tell how useful it actually is for their clients. To me it appears to be needlessly complex, but I believe these systems should be simple and useable above all else. Their marketing positions them as a significant competitor in this industry, but I’m not sure how much is hype and how much is truth.

Market-based (aka betting-based) solutions

Nosco IdeaExchange – Nosco is a great company from Denmark that I first met a couple of years ago. They first developed a portfolio of software applications that included prediction markets and what they call an Idea Exchange. Since then they’ve found much more demand for the Idea Exchanges and have since shifted their focus to that alone.

Their Idea Exchange is still modeled off of a futures market, where you can buy and sell ideas. I still would suspect a model like this to be susceptible to gaming and in general becoming a Keynesian beauty contest. (People don’t buy what is worthwhile, they buy what they think others will think is worthwhile.) That said, they’ve what looks to be a mature product that looks fantastic and has been used by a number of Danish companies.

Consensus Point – Some clients of ConsensusPoint use their standard ForesightServer software to run prediction markets on ideas. I’ve mentioned before how prediction markets aren’t suitable for this. To ConsensusPoint’s credit, they aren’t specifically marketing a one-solution-fits-all approach; it’s just what their clients are doing with the software.

NewsFutures Idea Pageant – I really like the quote from NewsFutures on their Idea Pageant page: “A large number of ideas makes a standard prediction market approach impractical.” While I don’t think that’s the only criticism, it’s a good chunk of a start.

The Idea Pageant is a fairly straightforward and easy to understand application. Each person gets a number of positive (green) votes/tokens and a smaller number of negative (red) votes/tokens. These can be refreshed periodically, and the consistently positive ideas float to the top.

Qmarkets – Qmarkets is another prediction markets startup that seems to have also moved into the innovation software arena. There’s a fairly extensive feature list; so much so I’m not sure how to gauge exactly how complex the software actually is. Without screenshots, it’s tough to tell how well developed the solution is, but it’s certainly a potential player in the market.

Xpree – The Xpree Open Innovation Markets seems to be a bit of a hybrid solution. It has a voting system for ideas, but then later management can transfer those ideas into a prediction market. This seems to be a well-thought-out way of approaching the problem. However, I personally believe that a client could shoot themselves in the foot with poor implementation of this software. It’s all too easy to start to turn every idea into a prediction market, and that (again) is a bad plan.

Marketplace solutions (specific innovations sought)

InnoCentive – InnoCentive is one of the most well-known idea marketplaces. They’ve had some good successes so far, and substantial press coverage. I would assume they are trying to quickly ensure they take advantage of network effects and become the primary marketplace for innovators and the companies looking for innovations.

NineSigma – NineSigma is another company in this arena. It’s less a true marketplace than a forum to receive and respond to RFP’s. They do seem to have a decent client list, and have been in business since 2000.

PhilOptima – PhilOptima is a similar innovation marketplace, but is aimed at “grant makers” and thus has a different audience on the innovation seeker side.

Innovation Exchange – This site appears to be a marketplace for innovations in general, without much focus. While that’s great in principle, I think the lack of focus perhaps hurts their chances in getting significant penetration in any market, and thus any substantial market share. I mention this because there will likely be a race for market share amongst these sites, and only the winner will get the ideal network effects.

fellowforce – This is similar to Innovation Exchange, but geared more toward the Web2.0 crowd, with widgets and rankings prominently promoted.

Full-fledged Marketplace solutions (specific innovations sought and sold)

Yet2 – I’m really intrigued by this site. What’s unique is that it offers something both for companies with specific innovation needs (like the category above) but also for entrepreneurs, engineers and scientists with innovations they believe have commercial potential. While the design is a bit harsh visually, it’s a very intriguing concept.

Other

Rite-Solutions – Rite-Solutions became quite well known a couple of years ago based on a well-known article in the New York Times that discussed how they used their software to allow everyone in the company to discuss and promote their ideas internally. It’s become quite a successful product for them (though perhaps not as lucrative as some of their government/gaming industry work!).

BrainBank – BrainBank looks to be a very interesting software solution that promotes both the ranking and refining of ideas, but also some management around implementation. It’s very interesting.

MindMatters – I classified MindMatters software in this category since I couldn’t quite tell what the main purpose of the software actually is. It mentions idea capture, challenges and workflow, but it wasn’t obvious how they all fit together in their particular software application.

Summary

There are a multitude of different approaches to innovation, and there are a multitude of different software applications to help companies and organisations innovate. Are there any true market leaders? Not as far as I can tell. Some, like Salesforce, are quite well known, but aren’t necessarily that useful for a wide cross-section of companies.

This post is meant simply to review and discuss different software applications available around innovation. I plan to write much more on how innovation does and can work in organisations. You can find all of my past innovation-related posts here, and future posts will go there, too.

I sincerely look forward to your feedback.

Reminder: If you’d like to receive future posts by e-mail, just click this link to get new writings delivered directly to your inbox! And if you use an RSS Reader you can click this link to subscribe.

I’ve had a couple of discussions recently about the general state of prediction markets; where the industry is going, where companies are finding successes and growth, etc. Some of these discussions can be difficult since there are several dimensions to a prediction marketplace, and thus the dynamics of what makes each successful (or not) varies.

So I am proposing a categorization method for prediction markets. Together, the criteria should be exhaustive and mutually exclusive. This could hopefully help describe the wide variety of potential prediction marketplaces.

Without further ado, here it is:

ICROP Criteria

• [I] – Interest: Horizontal, Vertical

  Horizontal indicates a broad focus, such as the exchanges run by Inkling, Newsfutures, and Hubdub. Vertical prediction markets are focused on a particular industry or sector, such as the Hollywood Stock Exchange, the now-defunct Storage Markets, or many enterprise prediction markets.

• [C] – Currency: Real-Money, Play-Money

  This should be straightforward. Is real-money or play-money used to trade on the marketplace?

• [R] – Rewards (maximum): Real (tangible) rewards, Intangible rewards

  Rewards should be classified by the maximum reward available on the exchange. It only distinguishes between tangible rewards, such as cash or gifts, and intangible rewards such as names on a leaderboard. So even Newsfutures and Hollywood Stock Exchange would be classified as Real Rewards sites, since you can win prizes on both exchanges. This is a fundamentally different potential motivation than a site like Inkling or Hubdub, where there are no tangible rewards available at all.

• [O] – Objective: Community, Market Results, or Background data

  This refers to the objective of the market creator. Is the purpose of the site to build or leverage a community, such as Inkling or the Chicago Sun-Times?

  Is the purpose to generate unique market results, such as Intrade or an enterprise prediction market?

  Or is the purpose to gather both the market results and information about the traders to generate additional background data known only to administrators, such as HSX?

• [P] – Privacy: Private, Semi-Private, or Public

  Again, this is largely straightforward. The semi-private case accounts for sites like Betfair that is generally public, but needs to limit participation for legal reasons.

Examples

These are how I would classify a number of prediction marketplaces based on the ICROP criteria.

Betfair:
I – Horizontal
C – Real-Money
R – Real rewards
O – Market results
P – Semi-Private

HSX:
I – Vertical
C – Play-Money
R – Real rewards
O – Background data
P – Public

Hubdub:
I – Horizontal
C – Play-Money
R – Intangible rewards
O – Community
P – Public

InTrade:
I – Horizontal
C – Real-Money
R – Real rewards
O – Market results
P – Semi-Private

Newsfutures:
I – Horizontal
C – Play-Money
R – Real rewards
O – Community
P – Public

[Acme] Enterprise prediction market:
I – Vertical
C – Play-Money
R – Real rewards
O – Market results
P – Private

Some initial thoughts

A few things strike me immediately. First is how so very few prediction marketplaces really utilize what I’m calling “Background data.” (This is something that Bo has mentioned before in reference to his prediction market work at Google.) There is a tremendous opportunity both in public and private markets to analyze and use trader data such as demographic information in conjunction with market data to create a completely new dataset. The only public site that I know of that has really done well here is the Hollywood Stock Exchange. But like Bo, I believe that this information could potentially be invaluable for corporate prediction markets. Unfortunately, none of the software solutions available have good feature-sets for this.

Second, there is certainly a link between Currency and Rewards. Real-money markets result in real tangible rewards, where play-money markets can result in either tangible or intangible rewards. Perhaps they shouldn’t be somehow combined, but I think they are important enough to list separately.

Summary

This isn’t a fully-formed classification scheme, and would certainly appreciate any comments (publicly in the comments or privately via e-mail) to refine it.

It seems that each month brings one or two new entrants to the prediction market software industry. (I can’t bring myself to use the term “space.”) There is now a lot of variety in the industry, with well-established companies, open source options for the ambitious, and new entrants that are pushing the envelope of what we consider a prediction market. As a consultant that works with a number of these companies to help meet my clients’ needs, I thought it would be useful to review the options as of November 2007.

This is my review of the different software available. Please comment below or contact me with comments, criticism, or corrections. All views below are mine and mine only.

I’ve tried to address software that you can license/download to use on your site, as well as public sites. This post is long, but is comprehensive.

Top Vendors

All of the following prediction market software vendors are well established in the industry, and have a background of successful clients and projects. I have personally talked to or worked with all of these companies, and would certainly recommend any of them. They are listed in alphabetical order.

ConsensusPoint - ConsensusPoint licenses their ForesightServer software, either as a hosted, on-demand solution or on a server within your network. Written largely by Ken Kittlitz, the software has evolved over a number of years and with feedback from a significant number of clients. A particular benefit is the potential separation between the front-end user interface and the back-end market software and database. Depending on how much time and effort you want to invest, you can completely design your own user interface for a custom solution.

Who to talk to – Dave Perry

Gexid - Gexid is a new software package, and has been used by Nokia in a project run by Bernd Ankenbrand, who just finished his PhD. I don’t know much about the software, but Bernd had some interesting results that he presented at the London prediction markets conference. From what I understand Gexid integrates a good chunk of consulting effort to complete a prediction market solution. Company established in Germany.

Who to talk to – Bernd Ankenbrand

Hollywood Stock Exchange - HSX is still the most prominent play-money prediction market out there. It, too, has over ten years worth of development effort. From what I understand, using HSX as your prediction market platform is most suitable for quite large-scale projects. Their past clients include big media prediction markets associated with magazines, television shows, etc. The back-end tools around data reporting and demographics on the system are particularly powerful, as they are the core of the main HSX business.

Who to talk to – Alex Costakis

Inkling - Inkling is one of the “new” kids on the block, having only been started in fall of 2005. (It was developed and partially funded through Paul Graham’s Y Combinator program.) Developed using the Ruby on Rails platform/framework, it has a slick user interface and can easily be rolled out for your needs. A particular benefit is their recent automatic free trial promotion… just sign up on their site and get going!

Who to talk to – Adam Siegel

InTrade/Tradesports - While InTrade and TradeSports are two separate companies now, the same software platform powers them both. They have occasionally licensed this platform for use by other organisations. Most prominently, the Financial Times used the InTrade platform to run the FTPredict site this summer (as discussed at the London conference here.) Though it can potentially be a bit expensive, the biggest benefit of this software is the “battle-hardened” nature of the platform.

Who to talk to – John Delaney

Newsfutures - Newsfutures is one of the few veteran prediction market software platforms, having been originally developed in 2000. They have a strong team, and have developed further related products for companies that don’t feel comfortable with a strict prediction market. (These include their competitive forecasting and idea pageant products.) The benefit of Newsfutures is that they do have these innovative ways of getting the same information in a structure that may be more appropriate for a particular company.

Who to talk to – Emile Servan-Schreiber

Nosco - Nosco has been around for about 18 months now, and has developed a great looking suite of software packages around prediction markets and collective intelligence. They talked about these at the London prediction market conference, and they include a news exchange, idea exchange, and traditional prediction market. So far they’ve had two top-tier clients in Denmark. They clearly have an excellent web design and graphic design team; everything they produced (slides, screenshots) were extraordinarily well done. Company is established in Denmark.

Who to talk to – Jesper Muller-Krogstrup and Oliver Pedersen

Pro:kons – Pro:kons has been in the prediction markets business for a decade, in some form or another. Their work now consists of prediction markets and new media projects. They’ve done some impressive work getting a prediction market platform rolled out for Swiss public television to predict the recent Swiss elections. In order to do this, they had to work with seven different television station identities and four official Swiss languages! They also presented at the London prediction market conference. Company is established in Austria.

Who to talk to – Gunther Fadler and Peter Gollowitsch

OpenSource Options

A number of open source prediction market software packages have been developed over the years. The software listed below is Beta-level or more stable.

Zocalo – Zocalo is written by Chris Hibbert, who was partially supported by CommerceNet during one phase of the development. It is certainly still in development, and you can contact him for more details.

IdeaFutures (aka ConsensusPoint, version 1) – IdeaFutures is the original version of ConsensusPoint’s software. When the ConsensusPoint ForesightServer software was completely rewritten, the previous version of source code was open-sourced. Though clearly not as fancy as the current version, this is certainly a stable and well-used platform.

Serotonin – I’m not terribly familiar with this piece of software, but it is listed as a Stable/Production-level in SourceForge.

MarMix – MarMix is based on USIFEX, which was previously developed by Peter McCluskey. It’s listed as Beta-level in SourceForge.

USIFEX – USIFEX is a prediction market platform that is no longer running, but does openly provide the source code at the link above.

Idea/Innovation Management specific software

These companies/solutions don’t offer prediction market software, but do offer solutions that are close cousins – software for idea selection.

Rite Solutions – Rite-Solutions “Innovation Engine” software was featured in a New York Times article last year. While not strictly a prediction market, their software certainly embodies collective intelligence and crowd wisdom. Employees can submit new ideas, help fund ideas with play-money, and also help move the ideas ahead by completing small tasks that are necessary to move the project forward. In addition to an Idea selection software, it’s really an Idea Management software.

InnovateUs – Again, InnovateUs is not a traditional prediction market software, but uses funding and voting to help identify new ideas out of a pool. They have at least one great client, and are based out of Chicago.

New prediction market software sites

Some of these companies provide private markets for organisations, and some just maintain a public site. I’ve tried to detail the differences below.

PrediCom – PrediCom has been working with prediction markets for a few years, and they have their own platform. They don’t seem to do much work here, and their software is always tied into their consulting services.

Xpree – Xpree was started this summer by Mat Fogarty, who previously championed and ran prediction markets at Electronic Arts using Inkling. They have gotten a beginning version of their software running that you can trial here. Xpree is focusing on a solution that is extremely simply for people to use, to help encourage maximum participation in a business environment. (Mat discussed this at the London prediction market conference.)

Nimanix – Nimanix is a new prediction market software company headquartered in Tel-Aviv. They maintain a (lightly-traded) public market, and are actively selling their solution to business clients. Their software appears to be a strict CDA model, with no market maker.

Qmarkets – QMarkets is a new software company, and is headquartered in Tel Aviv. It is very similar to the Inkling model, with a public marketplace and quick & easy setup of subdomains on their network. The site is quite new, and there is currently little activity on the public site.

AskMarkets – AskMarkets has been developed by George Tziralis in conjunction with his PhD thesis. While currently in private beta, it will hopefully be rolling out soon.

Foresight Markets – Foresight Markets is yet another prediction market platform. While their website is up, it appears that the software is still under development.

New sites, not prediction markets, but “Wisdom of Crowds” generally

GuessNow – GuessNow is probably the most prediction market-like of all these sites, and many people might call it so. You can choose which outcome of an event you believe is most likely, and most importantly, express your confidence level. The more confidence you express, the more points you win if you’re right and the fewer points you win if you’re wrong. (In their scheme, you always win some points just for participating, even if you lose.) My only problem with the site is the number and size of the ads… it’s a little chaotic to my eyes.

Predictify – Predictify has been in the mainstream blogs a bit recently with a couple of mentions on the Freakonomics blog. Unfortunately, Predictify really isn’t a prediction market… it’s a sophisticated polling and market research mechanism. That’s not to say it doesn’t use the “wisdom of crowds,” just that some of the assumptions that James Surowiecki talks about in his book don’t necessarily apply. The reason I say it’s not a market is that a user cannot express their confidence in their prediction. It makes it difficult to sort out the knowledgeable forecasts from the un-knowledgeable forecasts. Unique about Predictify is that companies can submit market research questions, and those that predict right can win the fees that the company pays to Predictify for the listing.

Foretal – Foretal is an interesting site, and has been running for just a couple of months. It combines a Free side, where you can vote (not trade) on what you believe will happen in an event, with a Free and Cash side. Predictions that are Free and Cash have the Free/voting system running concurrently with a Cash betting system that is essentially a pari-mutuel market. You choose how many 1 Euro contracts you wish to bet on whatever options are available. (They take a 5% commission.) On the legal side, Foretal says they are based in Malta, and don’t take cash bets from the United States.

BluBet – BluBet is another site that’s not really a prediction market. It somewhat resembles a pari-mutuel betting system, where you bet on a winner and the people who successfully picked the winner split the pot. However, in this case (like others discussed in this section) there is no way to express the confidence in your bet. If you are absolutely confident in your bet, you have no way to be able to bet any more than the person who chose a random selection.

ZiiTrend – ZiiTrend is really trying to integrate a prediction market and a social network. The prediction market software is quite slick, but stretches what it means to be a prediction market. While you can specify values, like you can specify the price at which you want to trade, there is no way to vary how much you want to risk. It’s simply a vote, but a variable vote. From what it looks like, they want to focus on the social network aspect of this site, and aren’t looking to sell/licence the software.

RIP – No longer available

CrowdIQ – CrowdIQ started around the same time as Inkling, but never seemed to get traction. You can still see elements of CrowdIQ around everywhere, as ConsensusPoint uses the CrowdIQ page layout/style in their default prediction market “look and feel.”

FreeMarket – FreeMarket was developed by Jesse Gillespie, but has since taken the software down. (See here.) Essentially even though he said he couldn’t support it, he kept getting quite a number of questions and requests for help. He’s moved on in his interests, so he just took the software down. That said, I have an old copy, so e-mail me if you’re interested.

Summary

Wow, there are a lot of options out there for prediction market software, prediction market public sites, and general “wisdom of crowds” (such as idea selection and voting systems.) Whether you’re looking for software to use internally in an organisation or you’re looking for public sites where you can try different types of interaction, there are many choices for you. I hope that the list above is useful for you.

As always, please feel free to contact me with questions or concerns. If you don’t want to miss any posts, why don’t you sign up for this blog via e-mail? Just click here and you’ll be all set.

I’ve recently read about a very unique software product, developed by a company in New York City. That product is called FogBugz 6.0, and was written by Fog Creek Software. The company is famous for the blog started by its founder, Joel Sposky, called “Joel on Software.”

FogBugz 6.0 is a software bug tracking and project management program that uses an innovative approach to forecasting ship dates based on what they are calling “Evidence Based Scheduling (EBS).” I think this software is likely very excellent at what it does. I also think that this type of feature is the perfect foil for how and why prediction markets can be used to forecast project management dates in companies and other organisations.

Prediction Markets and Evidence Based Scheduling provide the exact same core forecast data. A probability on when a project will be completed, and a plot showing how that probability has changed over time. Where they differ is in approach: bottom-up versus holistic.

Bottom-up approach:

Evidence Based Scheduling is a very data-centric approach. Each developer tracks their estimate to complete a given project segment. The software then provides a timer to measure exactly how much time was actually spent completing that segment. With that data in hand, the time it will actually take for a given developer to complete future segments can be calculated, with individualised standard deviations.

What the software requires is a list of project segments and who they’re assigned to. Monte Carlo simulations sum them all together to create the final probability curve.

What’s great about it: It’s data-based, and managers can dive down into the data to see why (and who) is the reason that a project may or may not make it out the door in time. Evidence Based Scheduling is also automatic, and doesn’t require any other interaction from the user to develop the forecasts. In a very data-rich task environment, this could be an excellent tool.

The problems with it: This works particularly well within its specific context, but I am skeptical that this would work elsewhere. Here are my concerns:

• Poor data / Garbage In=Garbage Out: The reason this implementation works is because each task is tracked and timed individually. The moment that the discipline around the process is lost, the data quality will suffer, and that will directly impact the forecast.
• Scope Creep: This method is based on having all tasks detailed. If a project suffers from scope creep, where additional features are consistently added throughout, the forecast is essentially meaningless. There are plots within this EBS to attempt to show this phenomenon, but the fundamental issue is still there.
• Can it deal with complexity?: I’m not convinced that EBS will deal well with projects that have lots of moving parts, or significant internal political issues. Political issues in particular can take completely de-rail well-run projects, and there is nothing in EBS to account for these complexities. (This is a bit of a combination of the two points above.)

Holistic approach:

Prediction Markets are another way to forecast when projects will key milestones. This can be done in a variety of ways, depending on your business needs. (You can find out more on prediction markets by watching these short videos: What is a prediction market? and How can I use a prediction market in my business?)

What’s great about it: Prediction Markets capture the holistic picture of a project. Instead of trying to forecast base-level data and then sum each part up, prediction markets forecast what you really want to know. Incentives encourage each user to express what they’re thinking, and their knowledge and history with similar problems at your company and in your industry. Even better, their level of activity corresponds to the depth of their conviction; if someone doesn’t feel they know enough they won’t trade heavily and those with heart-felt conviction will trade significantly more; it’s self-selecting.

The problems with it: Compared to Evidence-based Scheduling, prediction markets do have some issues. These are where prediction markets don’t do as well:

• No audit trail. While prediction markets provide forecasts, managers and executives can’t necessarily dive down into the data to see why they are changing. This can be mitigated by adding forums/discussion boards to the market, or providing a capacity to ask people why they traded after making a transaction.
• Less scientific “feel”. Evidence-based scheduling looks great because you can see and understand the building blocks of how forecasts are built. Prediction markets are based on people, their information and their incentives. Despite the fact they’ve been proven to work better than other forecasting methods, the “science” of prediction markets is based in economics, not in statistics.
• More work is required. Employees need to spend a few minutes of their time throughout the week/month in order to trade. EBS simply runs in the background (other than making the initial forecast of how long it will take to complete a segment of work). Time spent on a prediction market site can also be seen as a distraction by some managers, despite the valuable feedback it provides them.

Summary:

Evidence-Based Scheduling is a fantastic tool in its context; I just believe there aren’t many business problems where it can be effectively used. Prediction Markets use a holistic method to look at a project in total. While there isn’t as strong an audit trail in a prediction market, the full scope of the problem in the context of the company and the industry is taken into account. For many organisations, a prediction market will be the optimal solution.

Screenshot from FogBugz 6.0 website.

Cross-posted to Midas Oracle.