Mercury's Blog

Quick definition of innovation

First of all, there are multiple definitions of innovation and I want to address this. Here’s a (longer) definition:

Innovation is the economically successful introduction of a new technology or new combination of existing technologies in order to create a stepwise improvement in the value (compared to the resources invested) created for the client.

The problem here is with one word: stepwise. Some people and companies think this must be a massive step; others believe that fairly small step changes can still be considered innovations. In my opinion, this difference in understanding is why the word “innovation” has become the buzzword it has.

Innovation and tools in modern corporations

I’m interested in the intersection between organisational groups and innovation. Specifically, how can new technology help?

In a previous post I discussed the various idea management software packages available to businesses and organisations to help them innovate. But what continues to bother me about all of the software packages I’ve seen is that they seem to pass on responsibility. New ideas and innovations have to go through a process of review, which raises them to increasingly higher levels of management for further review. Employees themselves don’t keep the responsibility for success. They can help forecast and vote for an idea, but that’s it.

Perhaps my military background is showing a bit, but I think much more responsibility can be pushed down to the employees themselves. When I was in the Navy, there were clear rules under which we had to operate. But within those rules, we were very free to experiment and find the best solution for our watchteam/boat/squadron. This extended down to each individual watchstation; even the most junior enlisted man on board had room in which to learn and innovate. (This doesn’t mean that the Navy is an all-innovating organisation; just that there wasn’t needless process and structure for it.)

I personally believe that each additional step of process and each additional rule limits the boundaries of innovation in an organisation. Companies must operate with rules: spending limits are musts, managers must approve formal product introductions, etc. But these are rules for the firm, not for innovation. If you start putting rules and structure around innovation, (such as each project must have a sponsor, projects must have certain approvals before they begin, etc.) a company starts down the slippery slope to irrelevance.

What matters with innovative ideas is that they get implemented. (Or at least implemented enough to “fail fast”.) Does it really matter how if they comply with the main rules in the firm? What matters is results.

Bob Sutton is a very well-known management thinker and professor at Stanford. He writes here:

innovation often happens despite rather than because of senior management, and oddly enough, the best leaders often realize that their very presence can sometimes stifle innovation.

and a fantastic story (confirmed to be true) from HP:

Some years ago, at an HP laboratory in Colorado Springs devoted to oscilloscope technology, one of our bright, energetic engineers, Chuck House, was advised to abandon a display monitor he was developing. Instead he embarked on a vacation to California —stopping along the way to show potential customers a prototype of the monitor. He wanted to find out what they thought, specifically what they wanted the product to do and what its limitations were. Their positive reaction spurred him to continue with the project, even though on his return to Colorado, he found that I, among others, had requested it be discontinued. He persuaded his R&D manager to rush the monitor into production, and as it turned out, HP sold more than 17,000 display monitors representing sales revenue of $35 million for the company.

What to do?

Fundamentally, there is a difference between coming up with the ideas and innovations and formally developing them. At what point do you make what’s really an entrepreneurial-type activity a big-company project management process? Do you do it while it’s still just an idea, or later in its life? I know where I personally come down on this question… what about you?

Future prediction markets news

I’ve been working with a media company in London to develop a public prediction market for their (industry vertical) network. It’s still softly launching, and I don’t want to steal any thunder until they’ve had a chance to fully promote it. But I look forward to discussing it in the future here.

One last note…

As I mentioned before, this blog recently achieved a Google PageRank of 6/10. Because of this I’ve been getting a LOT more spam in the comments, and have made the comments section completely moderated. But please comment below; I will approve it (hopefully) shortly thereafter.

Happy New Year to everyone! I hope that 2009 is happy, healthy and prosperous for you. I specifically wanted to thank you as a reader of this blog, and those who have linked to my posts here. Google recently refreshed their PageRank for Mercury’s Blog, and it now has a PageRank of 6/10! Though I don’t post often, I do try to write long-form news and analysis, and I’m glad you’ve found it useful.

What I expect for prediction markets in 2009

I’m probably going to regret doing this by the end of the year, but I’m going to put on my forecasting hat and try to predict what will happen this year in the field of prediction markets.

• Prediction markets in 2009 are going to become even more well-known and wide-spread, but there will be no single event that brings them to the attention of the public. It’s going to be a slow, but steady, growth.
• All of the prediction market vendors will mature their business offering/proposition. Both InklingMarkets and Xpree have made recent business development hires, and Inkling recently redesigned their website with a stronger business pitch and case studies. I think that business cases for prediction markets will be more clear and more developed in general because of this.
• HubDub will continue to only be the only strongly popular play-money prediction market. (HSX is probably similarly popular, but is a single-industry prediction market.) With Hubdub’s partner program and social networking features, they will see significant growth this year.
• While a couple additional software vendors may appear, I get the feeling that the market for prediction market software is largely saturated. People working on projects part-time and selling them may proliferate and be suitable for the lower end of the market, I don’t see many more serious prediction market software companies starting up in 2009.
• I’m looking forward to see how the CantorExchange develops. Will there be enough customers? Will the market see the effects of other studios trying to bid the prices of their competitors’ films down? Or will it be seen more as a financial hedging tool for institutions involved in the film industry? There are lots of open questions to evaluate here, but I still maintain it’s a great step forward for the prediction markets industry.

What to expect from this blog in 2009

I’m going to continue posting on both prediction markets, but continue talking more and more about how collective intelligence can be used in other contexts. Specifically, I’ve become really interested in how groups of people develop new ideas and new innovations.

As I’ve written before, I really don’t like the word “innovation.” Bruce Nussbaum at BusinessWeek has written a few things lately that I really like:

“Innovation” died in 2008, killed off by overuse, misuse, narrowness, incrementalism and failure to evolve. It was done in by CEOs, consultants, marketeers, advertisers and business journalists who degraded and devalued the idea by conflating it with change, technology, design, globalization, trendiness, and anything “new.” It was done it by an obsession with measurement, metrics and math and a demand for predictability in an unpredictable world.

and

“Innovation” is inadequate as a concept to deal with these changes. You have “game-changing” innovation, which is big but rare and incremental innovation which is small but common.

The truth about innovation is that it takes groups of people a lot of work to find and develop new ideas and turn them into innovations. Scott Berkun has a great chapter in his book “The Myths of Innovation” where he destroys the myth of the lone creative genius.

We know it takes groups of people to make innovation happen, so where are the tools for it? A lot of time, money and effort has gone into social networking software to connect people, such as Facebook, MySpace, etc. But I have to agree with Tim O’Reilly: we need to stop throwing sheep and do something worthy. But those tools can be very useful to us in helping foster innovation. I reviewed the software for idea and innovation software recently, but was left generally unimpressed.

These are some of my current thoughts. I plan on thinking out loud more in 2009 on these topics and seeing where it takes me. I look forward to your feedback.

So a number of interesting things have come across my desk recently, all of which I wanted to write about but none of which really became its own post.

Askmarkets.com has launched!

George Tziralis‘s new project, Askmarkets, has now officially launched. George has been beta-testing it for a while, so it’s great to see it officially go live.

The most unique thing about Askmarkets to me is that it uses David Pennock’s DPM (Dynamic Pari-Mutuel Market) as an automatic market-maker instead of Robin Hanson’s MSR (Market Scoring Rule). It largely operates the same way, though with a DPM the person/company running the market isn’t required to subsidize the market-maker losses. (This theoretically makes it more attractive for eventual real-money applications.)

Nostradamical.com has launched

This is a new prediction market site that I heard about recently. It seems like a decent site, and I really like that it was developed by just one guy (Brad Young) here in the UK! While you can show your confidence in a particular outcome through a slider, I don’t like that I can’t adjust how much I “bet” on a given outcome. While I might be very confident about something in my industry, I wouldn’t want to bet the same amount on a pop culture topic. (I’m likely the last to know any Britney Spears news.)

Good luck to Brad as he improves and expands the site!

CFTC new scope?

The CFTC may be gaining a much broader new scope. If Senator Harkin passes his bill, it “would force all over-the-counter derivatives, including credit-default swaps, onto regulated futures exchanges.” This would be overseen by the CFTC.

To quote some interesting lines from the Wall Street Journal article:

[...]those financial contracts would be standardized and centrally cleared. It would also mean that they could be subject to CFTC-imposed speculative position limits and federal reporting requirements.

Under current federal law, certain financial derivatives are excluded from the definition of a commodity. The law, which was amended in 2000, created distinctions between what are known as “exempt” and “excluded” commodities.

The CFTC has no regulatory authority over excluded commodities, which include credit-default swaps. Transactions involving exempt commodities, such as metals and energy, may not fall under some of the CFTC’s rules and regulations if those contracts meet certain requirements.

Sen. Harkin’s new bill, dubbed the Derivatives Trading Integrity Act, would eliminate the distinctions in the law between excluded and exempt commodities, treating them all the same.

Hedging versus Gambling

Speaking of the CFTC, I was talking with some people recently about how to define the difference (in more layman’s terms) between hedging and gambling. The central question was: is there an inherent risk?

The classic case is that of farmers having a risk of their crop prices changing dramatically during the growing & selling seasons. The Chicago Board of Trade helped farmers hedge their inherent risk of significant market changes between the time they planted their crops and when those crops could be sold.

That can be contrasted to sports, which are typically understood as gambling. While some individuals may have emotional risk tied up in a teams’ win or loss, there is no inherent risk… sports are entertainment first and economic enterprises second.

In the middle, however, are contracts on something like the US Presidential election. A change in government ripples across the economy, and so businesses and individuals may very well have some risk in various election outcomes.

So while the question “is there an inherent risk?” helps to clarify some situations, there is still a spectrum of answers. While this may be second nature to some or many readers, I think it’s a better question to help separate hedging from gambling in many novice/lay-person’s minds.

Google Prediction Markets case at HBS

Andrew McAfee is a professor at Harvard Business School who does a lot of work, research and teaching on “Enterprise 2.0″. He has worked with/talked to Bo Cowgill at Google and has written a case study on Google’s prediction markets.

He’s also a twitter user and I wanted to quote one of his “tweets” here:

Google prediction markets case teaches like a dream – http://snurl.com/5q78q

I personally think this is a really good, solid step towards recognizing prediction markets as a valid and valuable internal tool. Hopefully more and more people will come into business knowing that prediction markets exist and that they can provide valuable information to managers about what their company is doing.

My HubDub post-election market

Finally, I put up my own post-election market on HubDub, on US Supreme Court justices. Please check it out below and trade!

How many US Supreme Court Justices will retire by the end of 2009?

There is a huge mis-conception in the media when it comes to evaluating the success of prediction markets in the recent US election. Simply put, you have to be wrong to be right. But depending on the type of prediction market you are operating, different methods of assessment are required. I’m going to go through each type of market and compare it to the best-of-class of poll aggregators (fivethirtyeight.com), discuss what they came up with, and potentially find a potential winner from the 2008 forecasts.

Probabilistic predictions (and prediction markets)

The most popular prediction markets this election season were probabilistic markets, where the payoff was either 1 or 0. (In the case of InTrade, $10 or $0.) The market price can and is interpreted as a percent probability that the contract will take place; one of the candidates getting elected. Unfortunately, many commentators believe that once a contract is above 50% that the candidate will win, and if a candidate is above 50% in a given market and doesn’t win, the prediction market has failed.

This is completely wrong.

I’ve discussed in previous posts that you have to be wrong in order to be right. As a quick reminder:

• 98% chance = 1 in 50 will be wrong
• 90% chance = 1 in 10 will be wrong
• 80% chance = 1 in 5 will be wrong
• 75% chance = 1 in 4 will be wrong
• 67% chance = 1 in 3 will be wrong
• 50% chance = 1 in 2 will be wrong

So how did the prediction markets do? And how did fivethirtyeight.com do in comparison? (They’re the only site I know of that takes poll results and turns them into probabilistic forecasts mathematically.) Here’s a quick snapshot:

InTrade results

We immediately run into the primary problem with evaluating these markets… while there are 50+ markets, only a small number of these were not at the extremes of the scales. I’ve just included those with a market price of <90%.

This looks great for InTrade, but it's deceiving. For example, let's take a look at all contracts at approximately 80%. (FL, OH, VA, NV, and CO). They have an average market price of 82.6%. If InTrade was perfectly calibrated, one of these should have been wrong.

It looks better on the other side. If you calculate MT, GA, ND, and IN they have an average market price of 26.75%. In fact, one of these four were wrong (Indiana).

For the mathematical amongst you, it should be clear that this type of assessment is quite crude. When you have so few data points, you need to pick and choose "bins" of data with your own best judgement. Ideally there would be enough markets that you could create strict "bins" of data and measure against those. (This is exactly how Inkling created their plots here… I would encourage you to read their post, too.) Unfortunately, we only have a small number of “battleground” markets and this just isn’t possible.

Fivethirtyeight.com results

In my opinion, fivethirtyeight.com appears to have done marginally better. Take the bottom five markets shown, which have a combined probability of 15%. In fact, one of these five actually occurred; a 20% success rate.

Another slice of data shows the same thing: MT, ND, IN, and MO have an average probability of 30% and an actual success rate of 25%. Taking MO, NC and FL the probability was 61%, and and actual success rate of 67%.

I say that fivethirtyeight.com has done marginally better because I could take reasonable slices of data throughout their predictions and come up with reasonably calibrated results. I had to specifically pick and choose to find similar results with InTrade. In other words, the fivethirtyeight.com forecasts were more internally consistent.

Non-probability predictions (and prediction markets)

Some of the lesser-cited and more lightly-traded prediction markets for the 2008 election cycle were on index markets on the vote share for each candidate. Here was the final tally of the national vote:

• Democrat – Obama – 52.7%
• Republican – McCain – 46.0%
• Other – 1.3%

What did the Iowa Electronic Market forecast? (Data from midnight on the 3rd)

• Democrat – Obama – 53.5% (0.8% error)
• Republican – McCain – 46.4% (0.4% error)

What did fivethirtyeight.com forecast?

• Democrat – Obama – 52.3% (0.4% error)
• Republican – McCain – 46.2% (0.2% error)
• Other – 1.5% (0.2% error)

To be fair, fivethirtyeight.com was the best of the “poll aggregators”. Real Clear Politics and Pollster.com came up with the following:

• Democrat – Obama – 52.1%, 52.0%
• Republican – McCain – 44.5%, 44.4%

It’s clear from this data that while the Iowa Electronic Markets were quite accurate, fivethirtyeight.com forecasts were even more accurate.

For reference, redbluerichpoor.com showed the following result from fivethirtyeight.com’s final forecasts (of state vote share) which were pretty accurate:

The issue of time

Recently, George Neumann of the Iowa Electronic Markets sent out a document to the Prediction Markets Google group that claimed that the IEM “continue to dominate polls.” One specific line really struck me as ridiculous:

During this 886 day period the average absolute error was 1.2%, amazingly similar to the final polling results but for a much longer period.

So now we’re supposed to assess the accuracy of a prediction market over a 2+ year period?!? So as the race moves and swings, and the markets with it, the IEM seems only to be concerned about the average. This implies that the election is much more static than I believe it to be.

Face it, the race changes. If the election was held within a week of the Republican convention, John McCain would have likely won… the markets reflected that. But events change, and the market changes along with them. This is why I disagree with Peter McClusky’s analysis of price changes here.

Uncertainty is priced into prediction market prices; in the fall Nate Silver of fivethirtyeight.com was consistently predicting a much higher probability of Obama winning the race than prediction markets, because the uncertainty of events between the forecast date and the election was priced into the market. As the election date got closer the uncertainty was removed from the price and Obama’s price went up. But this happens rather late in election futures; in election after election we’ve seen examples of late-breaking news that has had the ability to shift the outcome of a race.

Where does this leave prediction markets?

Were prediction markets consistently better than a well-performing site like fivethirtyeight.com? No.

Were prediction markets consistently worse than a site like fivethirtyeight.com? No.

They perform largely the same, though the final accuracy of fivethirtyeight.com was a tad bit better than InTrade/IEM. But the purpose of the two types of sites are different.

Fivethirtyeight.com and similar sites take current data and process it to extrapolate trends. These sites lag real events; Nate Silver mentioned on a number of times that he expected his model’s forecast to move, but that it hadn’t because the relevant polls hadn’t hit the model yet. Between the time a poll closes, the result released and then incorporated into the model is anything from a day or two to several days. So while it looks quite accurate, a poll aggregator is a lagging indicator. (Another couple of election cycles will tell us if their accuracy continues or was just a fluke in 2008.)

Prediction markets show the results of what a group of traders believe what will happen. This includes polling data, but also reacts to real-time information. A candidate makes a huge gaffe, and the market price will reflect it in minutes, where a poll aggregator could take days to see any effect.

This is the “social utility” of prediction markets, to answer a question that Chris Masse always poses. While they are on par with the accuracy of the best poll aggregators, their forecasts are real-time and reflect the state of the race right now. No other mechanism does this. While markets are certainly fed by polls, that isn’t the whole puzzle in and of itself.

Prediction markets worked quite well again this election cycle. Though their final forecasts were on par with the best poll aggregators, their real-time forecasts throughout the election season is the reason why they should be examined and discussed more broadly.

Tomorrow is going to be a landmark day for prediction markets. The 2008 US election cycle has been the most-polled, most-predicted, and likely the most-analyzed election in history. It’s been going for nearly two years, and I for one am glad the election will soon be over and governing (by whomever wins) will soon begin.

But why will it be a landmark day for prediction markets? Simply put: the data.

Prediction Markets and Polls – The Data

There are prediction markets on a wide variety of sites, with both play-money and real-money incentives. Iowa Electronic Markets, InTrade, and Betfair for real-money; HubDub, Inkling, NewsFutures for play-money. There are more, but these are the sites I’ve seen cited most often. (It’s too bad ConsensusPoint didn’t push TheWSX.com this election cycle.)

More importantly, there are a few sites that offer incredibly deep (and also probabilistic) analysis into polls. Most notably fivethirtyeight.com, which I seem to be checking a couple of times a day, now. There are national polls, national tracking polls, state polls, and even some state tracking polls! Fivethirtyeight in particular does deep-level statistical analysis to determine from poll results and demographic data how likely each state is to vote for each candidate.

The number of data points, from different prediction markets, polls, and poll trendlines/analysis will be immense. The sum total of data that will be available after this election should be a treasure trove for researchers, and should finally prove the accuracy of prediction markets.

But there’s a hitch…

Yes, there’s always a hitch, and it’s something I’ve discussed before. In elections, polls and prediction markets are measuring two different things.

Polls are measuring the percentage support for a candidate. Generally around 40-60% or so, unless it’s a total blow-out.

Prediction markets measure the percentage chance that the candidate will win their election. When the election is tight, around 50%, when it’s a blowout can regularly be 95%+. (Few prediction markets exist for candidates’ vote share; really only on the presidential level.)

What should you expect on Election Day?

I expect that a number of news outlets will be quoting percentages from InTrade in the run-up to the end of polls closing in the evening. Already final results contests are springing up, including one in the New York Times where you earn points based on current InTrade odds. You can also expect a LOT of volume on the markets tomorrow. But once the results start rolling in, the news is going to focus on the candidates alone. Wednesday will start the morning-after evaluation of the polls and markets, which will likely last for quite some time.

What does this mean in the end?

Comparing the results of polls and prediction markets is certainly like comparing apples and oranges. There are certainly some similarities, but they are fundamentally different.

What we need to do is evaluate how each forecasting method performed independently. For prediction markets, that means that a “failure” (where a prediction >50% didn’t happen) is quite likely a success. For polls, that means that a result just a few percentage points off (outside its MOE) is a failure.

I believe that prediction markets will come out looking quite good in this election. They’ve already proven their worth to me; when poll results might indicate a close or tightening race in places, the prediction market magnifies the difference, and in many cases demonstrates the poll volatility is just noise.

In the end, are the results from the prediction markets useful? Based on the number of times I’ve seen them cited this year… the answer is an unqualified YES. Are they perfect? No, and neither is any other forecasting system or technique.

I’m really looking forward to tomorrow…