Mercury’s Blog

If you're new here, you may want to get Mercury's Blog by Email or subscribe to my RSS feed. Thanks for visiting!

I’ve recently read about a very unique software product, developed by a company in New York City. That product is called FogBugz 6.0, and was written by Fog Creek Software. The company is famous for the blog started by its founder, Joel Sposky, called “Joel on Software.”

FogBugz 6.0 is a software bug tracking and project management program that uses an innovative approach to forecasting ship dates based on what they are calling “Evidence Based Scheduling (EBS).” I think this software is likely very excellent at what it does. I also think that this type of feature is the perfect foil for how and why prediction markets can be used to forecast project management dates in companies and other organisations.

Prediction Markets and Evidence Based Scheduling provide the exact same core forecast data. A probability on when a project will be completed, and a plot showing how that probability has changed over time. Where they differ is in approach: bottom-up versus holistic.

Bottom-up approach:

Evidence Based Scheduling is a very data-centric approach. Each developer tracks their estimate to complete a given project segment. The software then provides a timer to measure exactly how much time was actually spent completing that segment. With that data in hand, the time it will actually take for a given developer to complete future segments can be calculated, with individualised standard deviations.

What the software requires is a list of project segments and who they’re assigned to. Monte Carlo simulations sum them all together to create the final probability curve.

What’s great about it: It’s data-based, and managers can dive down into the data to see why (and who) is the reason that a project may or may not make it out the door in time. Evidence Based Scheduling is also automatic, and doesn’t require any other interaction from the user to develop the forecasts. In a very data-rich task environment, this could be an excellent tool.

The problems with it: This works particularly well within its specific context, but I am skeptical that this would work elsewhere. Here are my concerns:

• Poor data / Garbage In=Garbage Out: The reason this implementation works is because each task is tracked and timed individually. The moment that the discipline around the process is lost, the data quality will suffer, and that will directly impact the forecast.
• Scope Creep: This method is based on having all tasks detailed. If a project suffers from scope creep, where additional features are consistently added throughout, the forecast is essentially meaningless. There are plots within this EBS to attempt to show this phenomenon, but the fundamental issue is still there.
• Can it deal with complexity?: I’m not convinced that EBS will deal well with projects that have lots of moving parts, or significant internal political issues. Political issues in particular can take completely de-rail well-run projects, and there is nothing in EBS to account for these complexities. (This is a bit of a combination of the two points above.)

Holistic approach:

Prediction Markets are another way to forecast when projects will key milestones. This can be done in a variety of ways, depending on your business needs. (You can find out more on prediction markets by watching these short videos: What is a prediction market? and How can I use a prediction market in my business?)

What’s great about it: Prediction Markets capture the holistic picture of a project. Instead of trying to forecast base-level data and then sum each part up, prediction markets forecast what you really want to know. Incentives encourage each user to express what they’re thinking, and their knowledge and history with similar problems at your company and in your industry. Even better, their level of activity corresponds to the depth of their conviction; if someone doesn’t feel they know enough they won’t trade heavily and those with heart-felt conviction will trade significantly more; it’s self-selecting.

The problems with it: Compared to Evidence-based Scheduling, prediction markets do have some issues. These are where prediction markets don’t do as well:

• No audit trail. While prediction markets provide forecasts, managers and executives can’t necessarily dive down into the data to see why they are changing. This can be mitigated by adding forums/discussion boards to the market, or providing a capacity to ask people why they traded after making a transaction.
• Less scientific “feel”. Evidence-based scheduling looks great because you can see and understand the building blocks of how forecasts are built. Prediction markets are based on people, their information and their incentives. Despite the fact they’ve been proven to work better than other forecasting methods, the “science” of prediction markets is based in economics, not in statistics.
• More work is required. Employees need to spend a few minutes of their time throughout the week/month in order to trade. EBS simply runs in the background (other than making the initial forecast of how long it will take to complete a segment of work). Time spent on a prediction market site can also be seen as a distraction by some managers, despite the valuable feedback it provides them.

Summary:

Evidence-Based Scheduling is a fantastic tool in its context; I just believe there aren’t many business problems where it can be effectively used. Prediction Markets use a holistic method to look at a project in total. While there isn’t as strong an audit trail in a prediction market, the full scope of the problem in the context of the company and the industry is taken into account. For many organisations, a prediction market will be the optimal solution.

Screenshot from FogBugz 6.0 website.

Cross-posted to Midas Oracle.

This entry was posted on Friday, November 2nd, 2007 at 11:43 am and is filed under General, Software.