Tuesday, January 3, 2017

Why Scrum Sometimes Works And What Can You Do About It?

Scrum sometimes works great and other times is a constant source of dissatisfaction for both developers and management. If Scrum is working well for you, most likely you have already started modifying the "rules" of Scrum to fit your context. On the other hand, in a failing Scrum implementation, you are either looking to give up on it or are about to bring in a "Scrum Expert" that can help you course correct. The first reaction from your new expert friend is most likely going to be - "You are doing Scrum wrong". Scrum has some pretty straightforward rules which, admittedly, are easy to get wrong. There is probably a very low percentage of teams that adhere to all the rules in the scrum guide. In fact, in my personal experience, there is little correlation between adherence to the scrum framework and the degree of success of teams.

Note: The intent of this post is not to say that Scrum is a bad methodology. It is to say that doing Scrum "by the book" is very hard. Scrum itself has brought many great things to software development, as acknowledged here. This post tries to point out that those great things are available even without a full-on adoption of Scrum. 

That does not change the fact that many teams do see improvements when Scrum is introduced. What are the reasons that this improvement happens? Also, if success and improvement, have little correlation with the degree to which the scrum framework is implemented, can the same improvement be seen without ever implementing Scrum? Below is my take on some of the reasons teams see improvements with Scrum and how you can get there with or without Scrum.

Limiting Work In Progress

Scrum forces your teams to concentrate on fewer work items. There are only a certain number of stories that can fit into the sprint. This forces a team-wide work in progress limit. The team is able to concentrate on the few items in the iteration. This is usually a huge shift from each developer having 20 work items active with them. Any developer would tell you how costly this is. There is constant context switching and since you are caught up in 20 items. As a result, none of them get done. If any stories do get done, they are not done with the quality that a decent developer would be proud of. Context switching kills both productivity and quality. Limiting the number of work items you are working on, helps with both these aspects. Reducing "work in progress" also has the direct result of every item in progress getting done faster. That, in turn, results in more things getting done on a regular basis. There is a mathematical theory behind this. If you are interested in further details on the math, please read more about Little's Law here.

Here is the fun part - The benefits of limiting work in progress, do not have to come through Scrum. Simply having a rule that no developer can ever work on more than one work item (or even less than one) will produce very similar if not better results. This is not an easy change for most organizations. Scrum does not prescribe it explicitly either. This could be the reason why many Scrum implementations don't achieve the success they are expected to. If you are considering going Agile, this is the one single change I would start with. Explicitly limit the number of things your developers are working on.

Small Batches

The entire idea of having time-boxed iterations in Scrum forces the team to think of the work they need to accomplish in incremental small units. The Scrum Guide encourages these to be units that are close to a day in length. This can often mean that the limited amount of work that the team is taking on, is further broken down into smaller batches. Small batches have great benefits. They help find mistakes early, whether in the requirements, the code or the tests. They avoid big up-front design and heavy architecture work that often ends up as waste. Instead, design and architecture emerge as new needs are discovered. The team continually makes measurable progress towards its goals rather than having little idea of where they stand in the overall picture. Small Batches help gain a lot of predictability. Most successful scrum teams usually reinforce this by saying that they will not work on anything that higher than 5 or 8 story points. This is not a prescribed rule for Scrum, but one that seems to be commonly used by successful Scrum implementations.

Interestingly enough, small batches are completely achievable without adopting scrum. Developers can break down work items into batches that take 2-3 days to get done. Every time they pick up a work item, they can the question - Can this be done in less than 3 days? If the answer yes, they start work, otherwise, they break it down into pieces that are achievable in 3 days or less. Of course, they are allowed to be wrong, and some items will take longer. This approach will give you the same benefits of small batches with or without the adoption of full Scrum. If your current Agile implementation lacks the emphasis on small batches, that is another easy win.

Collaborative Improvement

Scrum uses its ceremonies, Planning, Daily Standup, Sprint Review and Retrospective as tools to establish collective ownership of the process and the product within the team. The most powerful of these, for long term improvement, is the retrospective. This is the activity where the team gets together at the end of an iteration and figures out things that are going well and things that can be improved. Scrum did a great job of taking the job of "figuring out efficiencies" away from managers and handing it to the teams. There are numerous retrospective techniques out there. As long as teams are looking to improve, you can employ most of these techniques and get successful results.

For some reason, teams adopting Scrum are forced into using the same cadence (sprint length) for Planning, Retrospectives, Stakeholder Reviews and delivery. There is no reason for these cadences to be intertwined. Scrum puts retrospectives at the end of iterations or sprints. They don't have to be this way. If you are not doing Scrum and don't have sprints established (and there might be no good reason to), you can do retrospectives as and when an issue that needs the team to get together pops up. This goes hand in hand with working in small batches.

Small batches work effectively whether we are talking about developing software or making improvements on a team. Instead of building up a batch of issues to talk about, let us take care of things as soon as they come up. This might mean that there is no established cadence for a retrospective and that, in my opinion, is perfectly fine. It might work better than having a backlog of painful items build up for two to four weeks. A large retrospective backlog can lead to ineffective retrospectives as not all the important topics can be talked about before people burn out. Retrospectives, themselves, are not unique to Scrum. You do not have to be doing Scrum in order for your team to improve collaboratively. Have retrospectives as and when you need, give the people doing the work, the power to improve collaboratively.

User Feedback

Central to the entire idea of Agile is to get feedback from users and stakeholders. Scrum does this by having a sprint review/showcase at the end of every sprint. This is what really puts the Agile in Scrum, especially if you are smart enough to make decisions based on the user feedback. The problem is that Scrum regiments the user feedback to the end of sprint, review step. It is the same cadence matching problem that retrospectives suffer from. The earlier we get this feedback the better, then why wait until the end of the sprint. Get the feedback as soon as you have something ready.

If you are working in small batches and limiting your work in progress, it is likely to have something ready for review every day. These changes should not have to wait till the end of a sprint in order to be shown to users and stakeholders. For a developer, 2 weeks is an eternity when it comes to remembering what he/she did. There is a great deal of efficiency in faster feedback. We can tweak something a developer just worked on before the dev has moved on to working on a different part of the system. Get feedback, course correct and deliver value as early and as often as possible.

The (Re)Starter Kit

Scrum is often touted and used as the starter kit for Agile. The problem is that Scrum while appearing simple on the outside is very hard to "do right". The reason people are attracted to Scrum as a starting point usually is due to the fact that Scrum is a documented recipe. Do these things and you will be Agile. Every Agile coach will tell you that just doing the steps does not make you Agile. Unfortunately, the same recipe is the reason why Scrum adoptions fail and need reboots. Developers are analytical beings. If they believe that the recipe produces the Agile cake, they will follow it in every detail. The focus shifts from the intent of the law to the letter of the law. A system designed to help developers quickly changes into a way of micromanaging developers.

Inflexible rules lead to inflexible processes. Inflexible processes by their nature are not great at adapting to your context. In order to be successful with Scrum, most of the times, you have to be flexible and tweak the rules to fit the context. Is it necessary though to have the rules in the first place? Why not have principles and let them define the rules in our context? The basic premise for Agile, in my opinion, is - Deliver early and often and use feedback to determine the future course you need to take. Let us take that premise and work with in order to make things better.

 Agile does not mean Scrum, although Scrum can at times be Agile

If Scrum is failing you, or if you haven't tried it yet, there might be a simpler way to dip your toe in the Agile pool. Start with the four principles here (or a subset of them), and measure the gains they get you. The rest of the Scrum framework is barely required if you want to be Agile. 
  • Limit Work In Progress
  • Work In Small Batches
  • Improve Collaboratively
  • Get Rapid User Feedback
I would argue you can be more Agile with these four changes than you would be if you adopted full on Scrum. Agile does not mean Scrum, although Scrum can at times be Agile. This is not to say that Scrum is a bad way to go. The point there are some background "intents of the law" that make Scrum work. It might be a simpler approach to adopt these intents in order to create an Agile mindset as opposed to adopting an Agile framework. Teams see improvement Scrum with not because they strictly adhere to Scrum rules, but because of the intentional or unintentional adoption of practices that make them Agile.

In the interest of small batches and of limiting your WIP, pick one of these four principles and start there. This might be more effective than taking on an entire framework. Each of these principles in themselves is not easy. Wouldn't it make sense to try one smaller difficult thing, rather than a set of multiple difficult things at the same time?