Monte Carlo. It’s a car. It’s a song. It’s a casino. It’s a city in Monaco (near France, somewhere). It’s also a method of statistical simulation that is used to better understand the probabilistic distribution of known set of data. Great. So what?
I recently attended the Atlanta session of the FogBugz World Tour to hear from Joel Spolsky about the latest version of his bug tracking software, FogBugz 6.0. Joel Spolsky has been writing about software development, management, business, and the Internet at joelonsoftware.com since 2000. Three books have sprung from content on his site: User Interface Design for Programmers,Joel on Software, and Smart and Gets Things Done. They are good reads for software developers and business folks alike; smart, funny, and focused on the human face of software development.
Before starting his company, Joel was a Program Manager at Microsoft on the Excel team to provide programmability to Excel.
The new release of FogBugz has quite a few nice features to make bug tracking much easier, but I was most intrigued by the new capability called Evidence-Based Scheduling, or EBS for short. This new capability uses a Monte Carlo simulation approach to determine probabilities associated with the expected “ship dates" of the software release project.
The core premise behind this capability is that unlike in the financial realm, in the software world, future results can accurately be based on past performance. FogBugz remembers the original estimate and the actuals for each task for each developer, and for all the projects they’ve been assigned to. Here’s where it gets interesting. Based on this information, FogBugz runs a series of Monte Carlo scenarios where it randomly generates different plausible results for each member of the development team. It then assembles the results to create a distribution for probability of completion for each developer—and more importantly, for the entire project. The result of this analysis is a "Ship Date" probability curve that shows potential ship dates on the X-axis and probabilities of achieving those dates on the Y-axis. Ship Date depends not only on estimates for remaining components, but also on the probability that the individual developers will be able to individually meet their commitments. A steeper curve means the developer estimates are more confident and a flatter curve means estimates are less confident.
The classic software development process involves balancing three things: time, money, and features. Most projects of reasonable duration are not going to be able to effectively add staff mid-stream—at least not to accelerate delivery timeframes. If you assume the primary factor in determining "money" is tied to people, time and features are your only real variables. FogBugz 6.0 lets you experiment with changing these two factors. This is a useful tool. Consider a scenario where the business people come to the development lead because they need a project to be complete by a specific date. This tool gives the project lead enough information to understand the probability of making a specific date. Additionally, it lets you test out what will happen if you remove a particular feature. With this new information, the development lead has the visibility to provide back additional guidance to the project sponsor about the probability of making the requested date, as well as the effect that changing release date and scope have on the probability of on-time delivery.
So how does it know? The ship dates are based on each developer’s history of estimation accuracy supported by developer time sheets. Yes, that’s right. I said time sheets—the bane of all developers. But stick with me, it’s not as bad as you might think. Each developer (or responsible team lead) can turn on a timer that automatically tracks time spent on each of their cases.
Fogbugz plots a linear fit through the data points for each developer and then uses the calculated R2 value to determine how consistent the developer’s estimates have historically been. Based on this calculation, probability distributions for remaining tasks for each team member are determined, which leads the ship date probabilities. It’s then easy to see the long pole in the project (Milton Ritchie in this example). This doesn’t necessarily mean the most work, but only reflects the probability of on-time completion—and correspondingly, the most risk to the project.
Anyone who has followed Joel’s writings knows that he is keen on the idea of improving the process of software development. And, we all know that estimating an accurate time to completion, or “ship date," for any software project is a pain in the rear even after all these years of "practice." The unique approach that Fog Creek takes is not to predict the specific completion time, but rather to predict the probability of completing the project across a range of dates.
So, project estimating will likely never be a simple turn and churn process. But with this release of FogBugz, Fog Creek Software has shown great outside-the-box-thinking that could significantly improve how we deliver software projects.