Thursday, June 30, 2016

The Audacity of the Strategy of Hope

Typing the words "Hope Meaning" into Google (since Google knows everything), gives you the following definition of hope -

Whether hope is being used as a noun or a verb, the implication for projects is the same. There is always a degree of hope involved in projects that are either being planned or are already active. Projects start with a certain level of confidence of hitting the desired end date. The point at which our confidence in making the date falls below our initial confidence is when we start employing the "Strategy of Hope".

A soccer team that is up 2-0 at half time is operating in the second half with confidence because they have a very high likelihood of winning the game. The percentage of games won from this position is very high. On the other hand, the losing side at the same point is operating (amongst other things) under the strategy of hope. According to, "a mere 1.8% of 2-0 leads have been squandered into defeats, whilst 93% have been converted into wins". Despite knowing these percentages, we still see the hope bias, both in sports and Software development. We always believe that we can be the 1.8% and not realizes that only 1.8% can be the 1.8%.

The language used by development teams and managers starts to change the more they start to rely on hope. Words that represent high degree of certainty like "definitely" and "will", are replaced by words that represent more hope than certitude. Phrases containing words similar to "if", "may" and "hopefully" become the norm. This lasts usually till the last day of the project, when the certainty comes back. This time though, it is "definitely not" and "will not" that are the phrases which are most commonly used. Hope turning to despair in the last throes of a software project is all too common an occurrence. 

I believe that traditional application of hope in projects follows the graph as shown below. The reliance on hope comes in multiple forms. The most common form, which I have been guilty of as well, sounds a bit like this - "We have been going through a rough patch due to [insert reason here], we expect to get over the hump soon and make up ground". It is usually not till the release date or the day before the release that we finally admit that there was too much ground to make up.

Saying that it looks bleak, but we will definitely get it done, is usually just delaying the point of despair. There are two things that commonly happen as we approach the point of despair. First, the team starts working overtime to get the impossible done. Second, the team takes shortcuts to get the work done. Both of these avenues are likely to result in quality issues and creation of tech debt. In circumstances where the date or scope is negotiable, the information regarding a delayed release or delayed features arrives too late, causing loss of credibility and possibly revenue.

How do we escape the hopeless Strategy of Hope (Alright, it is not completely hopeless, every once in a while it does work)? Whenever we see a project that is starting to rely more on hope and less on what the team's actual current performance tells us, it is time to self correct. Avoiding reliance on hope is not possible, there is always a degree of hope in every endeavour. If you have planned the project start and end dates knowing the confidence level you have of completing that project on time, and the confidence starts getting replaced by hope - that is where you readjust scope or time. Set expectations early and often.

We live in a probabilistic world. Every software professional will tell you they do not really know exactly how long it will take them to fulfill a single request and get it into production. I am not sure why we have the confidence to know exactly how long it will take to get numerous requests into production. Setting end dates using probabilistic forecasting measures and not being deterministic about a release date and release contents is accepting reality. Acknowledging reality and shedding the reliance on hope is not taking the easy way out, it is simply doing business properly. Acknowledge your reality early and acknowledge it repeatedly as often as you can.

We use Monte Carlo (more on these in another post) simulations to understand our reality when it comes to releases. We run these simulations for our releases every hour, for each team, to find out where the teams stand and what kind of progress they are making. The simulations give us a percentage probability of the team being able to finish all the items in the release by the release date, based on their recent performance. This means that we find out very soon whether our confidence in the completion of the release has dropped below an acceptable level. Invariably, when the confidence in the release goes below a desirable point, the Strategy of Hope starts to make its appearance. This is the point where we try to look for alternatives to hope. We look to realign expectations as early as possible rather than waiting for the last part of the release. Teams should aim to avoid disappointment brought about by despair at the end of a release.

Monte Carlo is not the only way to do this of course. Whatever is your metric/simulation/gut feel that shows that hope has become the operative method, use it to detect the point where you readjust expectations. Lower you tipping points for despair, so that you do not have to wait too long to realize that your expectations are not aligned with reality. Whenever your reliance on hope exceeds a certain point, realign expectations if possible. If realigning is not possible, be prepared to deploy the emergency fixes immediately after the release and carry the tech debt forward for eternity.

Strategy of hope is probably best left to soccer teams that are 2-0 down at the half. They are waiting for the miracle in the second half. They are carrying the "feeling of expectation and desire for" that miracle. It happens once in a while, that a team that has not scored a goal in a half, will score three in the second half, but do you really want to bet on that miracle?