Probabilistic Forecasting - The Flaw Of Averages

The Big Question

Whenever we are planning out a release, the first question that we try to answer is - What is the team's capacity? Similarly when we start work on a feature, the first or story question is - How long will this take? We have traditionally tried to answer these questions in multiple ways with very little attention to the fact that there are multiple possible answers. Each of these answers have their own probability of being correct. We need tools and systems that help us figure out what these answers are and what are their probabilities of being correct.

The Flaw Of Averages

Our traditional approaches to projecting capacity and of tracking progress have been fraught with errors. We have in the past used techniques like average number of stories, or average number of points or even stories done in the last release to determine what we believe is the capacity of the team for a given release. These approaches suffer from a few major flaws. The greatest of these flaws is that average is at best telling you that there is a 50% likelihood of doing the same amount of work or more and 50% of doing less. For greater level of detail into why "Plans based on average fail on average" please read the fascinating book "The Flaw of Averages". 

This calls for a better way to predict and track capacity. A method that does not just give us prediction in the form of a date or a number, but also the likelihood of that prediction being correct. We need something that is better than average, something that lets us know what is the level of the risk we are taking when we say that a team will be able to get X number of work items completed. Every prediction has the possibility of being wrong, hence, when we make a prediction, we should acknowledge what is the probability of that prediction being correct and conversely the probability of the prediction being wrong.

The Flaw of Averages, does not preclude us from using the past as a predictor of the future. Past performance is the best baseline we have to project future performance. There are better methods to do this than traditional methods like gut feel estimates and averages. A detailed look at our past performance can reveal the level of uncertainty that exists in our system. An Understanding of this uncertainty can give us the ability to make better predictions. Modes of forecasting that boil predictions down to a single number are flawed because they don’t take the random nature of software development into account. Before we get into modes of prediction, let us understand the nature of uncertainty in software development.

Next : Understanding Uncertainty