Probabilistic Forecasting - Understanding Uncertainty

The subtitle of the book The Flaw of Averages is - "Why We Underestimate Risk In The Face Of Uncertainty". Let us talk a bit about uncertainty. Uncertainty comes in two flavours - Single item and Multiple item. The Single item uncertainty refers to the variation in lengths of time around how long it takes for one thing to get done. An example of this would be the cycle time of a single story or cycle time of a single feature. Multiple item uncertainty refers to how long does it take a set of single items to get done. This would be equivalent to figuring out capacity of a team for the length of a release. In other words - How many stories can we get done in a given time frame?

Single Item Uncertainty

For those who commute to work - How long does it take for you to get to work? Let us say your answer is 15 minutes. Would you bet your paycheck on getting to work exactly 15 minutes after you leave home? If we started noting down the amount of time it takes every day to get to work, we would probably see a very random distribution of times(with a high likelihood that none of them is exactly 15 minutes). The reason for the random distribution of time, is that there are inherent uncertainties in the process of commuting from one place to another. Traffic lights, errant drivers, accidents, heavy traffic and many other variables come into play. All these variables, which compound on each other lead to the uncertainty that prevents us from knowing the exact time we will be at work. There are numerous parallels we can draw between commuting and getting a work item(story or feature) through our process and calling it finished. After years of having tried to do this, we know that asking a developer to estimate how long a story is going to take is a futile exercise. This is due to variability built into knowledge work. We do not know what we will find when we go to try and implement something, much less how long testing it would take. The best we can do is give a range that we are comfortable with. Even beyond that, there is no direct predictability on when a build with the change will become available, or how long a change will sit waiting for someone to pick it up to test. The greatest sources of uncertainties though, are external dependencies. Whether at story or feature level, completion of a work item that cannot be done by one team, is very difficult to predict.

The one thing we can say for sure is that the larger a work item is, the more unpredictable it will be. It is again analogous to a trip in the car. A "15 minute" trip can be variable by 10 minutes, most of the time, but a "4 hour" trip can easily be variable by 1-2 hours. When something looks like it will take a long time, it is more susceptible to variability and the completion time for it is increasingly uncertain.

Multiple Item Uncertainty

Imagine we were holding a chess competition to figure out the best chess player among a group of 10 players. The structure of the tournament means that we need to have 20 games in order to crown a champion. Each game can take anywhere from 15 minutes to 3 hours. We have 2 chess boards. How many hours do we need to schedule in order to make sure we have enough time to finish 20 games? This is the multiple item problem that we try to resolve everyday when we try to figure out how long a particular functionality or feature is going to take to develop. Looking at this another way - How many games can we play in a 24 hour schedule? That is the same problem as trying to figure out how many stories we can get done in a release. As you can see, the fact that each game itself can be of a variable length, makes it very hard to predict the completion time of all 20 games. We can easily be drawn into the flaw of averages thinking here. There are also dependencies between games. The earlier round games have to be played before games in a later round. Playoff games have to be played in a particular sequence. This would make our job harder. If we do not have players available at certain times when they are scheduled to play, it would add more delay to the tournament as well.

What makes this a hard problem to solve, is that there are multiple answers. The total time for the tournament (just like the car trip) is not just one number, but multiple possible numbers. The trick is to figure out what is the probability of each of those numbers. The easiest way to do this would be to play the tournament thousands of times and record how long it took every time, so that we can use that data to determining probabilities for different lengths of times.

Next : Forecasting Single Items