Why Your (Big Bang) Multi-Year Project Will Fail

If you are reading on a mobile device, try the Medium mirror of this post.

Imagine you have been tasked to produce a brand new product. You have been asked to create a project timeline and given high-level requirements. All initial estimates point for this to be a 3-year project. How would you go about developing this product? For the sake of this post, we will consider two different approaches to the same project. First, is the more traditional big bang approach. One where the deadline is set three years out. We know there is a lot of work that needs to get done and our best estimates tell us that this will be done three years from now. Second is the more agile, iterative approach. Here we believe that we can deliver some minimal functionality in 3 months, and push out updates every week or so. We still estimate that it will take us three years to build the initially requested functionality. Let us take a look at what these look approaches look like in real life. We will first play this out for the big bang three-year project and then the iterative one.

Big Bang

This is the approach where at the beginning of the project we know what we are going to build and we have estimated out that it will take us three years to build it. We are going to work through the next three years and ultimately deliver the entire project to the customer.

Product Direction 

A three-year plan has a very basic assumption built into it - We know everything that the customers want at the beginning of the project. Of course, we leave room for things coming up, but in general, we know the things we are going to accomplish over the next 3 years. These are the things that we have started to promise our customers. They will be available when the first paid version of the application ships. If the competition forces us to change our course or new compliance requirements come up, we either push the team harder to make the same date or we disappoint our customers. There is little room for surprises and every surprise is a threat to the ship date. As the project goes on, the competition comes up with new innovations. Someone inevitably mentions artificial intelligence or big data or meta-platforms or something else that we really need to be doing halfway into the project. Seeing that what seemed as the state of the art when the project was started is not the buzz anymore, these new concepts become part of the scope. The date cannot move much though because it has been promised. We ask the teams to go re-estimate and figure out how to fit everything into the timeline.

We might set up internal milestones for the teams, but engineers are a lot smarter than we give them credit for. They know that these internal dates are bogus. Especially when the internal dates are not met and still the final date doesn't shift. They quickly realize that the internal milestones are just mini death marches to ensure that the developers are moving as fast as possible. Managers love these fake targets as it helps them show progress and feel good about hitting them. The developers, though, pretty soon catch on to the smoke and mirrors when the end line is not moved despite having to work nights and weekends for these internal targets. Rarely are our initial assumptions ever challenged and things removed from the roadmap. Even if we have to go on death marches to meet internal dates, the scope and date for the final delivery are fixed.

Prioritization

Since we have 3 years to finish everything, there is no real priority on anything. It does not matter what we work on first because everything will be delivered at the same time. We pretend to have a priority order to give our project some structure. In fact, Project Managers have all the fancy portfolio management tools, excel spreadsheets, Gantt charts showing that there is an order in which these things should be accomplished and also showing that their jobs are justified. But these charts, in reality, mean nothing. There is very little customer validation of what is being built. No customer is expected to use this product till it is "finished". Which means it does not really matter if you work on workflow before any functionality that requires workflow even exists. 

Since everything is important and there is no intermediate delivery, multiple teams start multiple features at the same time. These include features that might be a very low priority for the project, but since we have the people available, and three years in front of us, why not? Everything needs to be done by the end of it anyway. This often results in multiple 80% done features as the scheduled date nears. This is the point where we start cutting corners on code and product quality. Starting multiple things without any real prioritization at once puts them all at risk.

We have started with the assumption that everything that we are working on is necessary to be done and at no point are we willing to question those assumptions. As long as all the features we said would get done are done, it does not matter which ones were the most useful. It also does not matter if we worked on something for multiple months and no one ever used it because it got delivered on time. We will probably never realize that no one is using it anyway since our goal was to ship the feature within the 3-year timeframe, not solve a customer problem.

(Big Up-Front) Design

As the company is making a huge investment by embarking on a three-year project, everything has to be perfect. There is a good amount of time spent doing up front analysis and design. Multiple months might be spent doing research even before any code is written. Most often a lot of this analysis is thrown out soon after we actually start writing the code. It is usually discovered that either the design was untenable or our initial customer validations of the actual screens told us that the analysis was wrong. Expert consultants might be brought in to help create the perfect modern architecture. This is where the Architecture group shines. The architects have the pressure of building the perfect architecture on which this 3-year plan is to be executed. An architect is a very dangerous animal. He or she is often a software engineer who has not written any customer facing software in a while. The Architect does not work on teams but instead designs and executes the architectural vision. This type of project is perfect for the architect. This is the opportunity to play with new patterns and to do green-field research. Architecture groups can take months to produce something and no questions will be asked because, in order to be successful, the product architecture has to be perfect.

The resulting architecture is cumbersome to work with as it tries to solve every imaginable problem the project would ever face. Solving for the future almost always ends up slowing down the present. Instead of the architecture evolving with the product and the customer's needs, it is running ahead in its own pursuit of perfection. We don't know, for sure, if these are actually problems that need to be solved. We only have assumptions that the customers are going to need these solutions, leading to premature optimizations. Per Donald Knuth - "premature optimization is the root of all evil (or at least most of it) in programming". Developers slow down tremendously as the architecture is continuously in their way. Instead of making development easier all the extra technical requirements have slowed progress to a grinding halt. It takes dozens of professional developers months to produce the same functionality a couple of college students could have produced in a week.

Premature optimization is the root of all evil (or at least most of it) in programming. - Donald Knuth

Developer Morale

Managers are baffled by the slow pace. Deadlines and promises are now at risk. We need to take the developers aside to re-estimate to understand the risk. We need to have meetings and surveys about how to fix things. The developers usually respond to these surveys honestly the first couple of times. They express their frustration with the process, the architecture, the way the project is being managed etc. Since this hits directly at the people evaluating the survey, the results are mostly ignored. A couple of appeasing words might be said to the developers, but we don't really have time on this project to make sweeping changes. We have already invested multiple months in our current strategy, we can't change it regardless of how bad the people on the ground think it is.

Developers then try to fix this by trying to work around the architecture or by trying to influence change in the architecture. They come up with proposals to make things go faster and they usually involve dumping many of the complicated architectural components. Suggestions might even involve switching technologies to ones that lend themselves better to the project at hand. These suggestions are given some thought but usually discarded. They threaten the reign of the architects and also the timeline at least in the short term. Developers cannot be made to think that they know better, otherwise the project will run out of control.

Most successful and driven developers like to see the products they build being used. Developers derive fulfillment not just from solving problems, but also by producing customer value. If a developer has to wait up to three years for the product to see the light of the day, morale will invariably decline. The flip side of the argument is that if we have developers that do not derive fulfillment from delivering customer value - Should we really have them on the team?

Getting feedback early and often helps the developers be connected to the users and results in software that is much better at solving business problems. It also results in developers who want to solve business problems and deliver these solutions as soon as possible. In the case of the three-year project, we are actively telling our developers to not care. We are telling them to put their heads down and keep plowing through using the frameworks and technologies they have been told to use. It does not matter whether what you worked on was useful or not, here is the next story or feature to start. Can you not see we have a deadline, we don't have time to stop and verify. We might even run an Alpha program that would not change the course of what we are working on by much.

(Long Term) Predictions

Almost all of these issues stem from the fact that right at the beginning of the project we assumed that we knew exactly what we want to do over the next 3 years. We have made predictions that are three years out and have started making bets on them. We have made promises that need to be delivered on. Managers have promised progress to directors, who have promised a product to the C-Level. The C-Level and sales team have gone out and already sold the product with a full feature set to the customers. There is a lot of disappointment to go around if this does not happen. Heads will roll. Our Gantt charts tell us that things are slipping but we believe we can make it. If we add four people and increase our per person throughput by 50% we will definitely make it. We are projecting multiple years in advance at the same time as we have trouble meeting our internal monthly milestones. Everyone on the team knows that the project is heading for disaster but they have been beaten back enough that they don't want to be the bearers of bad news. After all, heads will roll. We are trying to predict the impossible - How much can we get done in 3 years? Once we have made the prediction, we are sticking to it, because promises have been made and now we cannot back off of our predictions. The interesting part is, even if we build everything we said we would, there is no guarantee that we have actually built a successful product since it has never been tested in the field.

Ending The Project

The three-year project has a great cost involved with terminating it early. There is nothing to show as a result. We have a whole bunch of unfinished features that don't work cohesively together. It is not a product at any point of time before the very end. Exiting the project in the middle seems to only have costs and no benefits. We cannot sell a half-finished product and since we have spent all this money in building this half-product, we might as well bite the bullet and finish this. We are living smack in the middle of "sunk cost fallacy" land. The length of the project makes it very likely that we will have to make this decision. Market forces change rapidly. Assuming that we can hold a course for 3 years amongst these changes is irresponsible on the part of the management at any company.

Let us see how the same project if done in an iterative manner, progresses.

Iterative

In this case, we would assume that the project has the same initial objective - To deliver a product in 3 years. We are instead going to take a different approach. We are going to aim to deliver a much smaller feature set in 3 months. We will then deliver incremental bits of functionality as often as it is completed (every month, week or day as appropriate).

Product Direction

Product direction, in this case, starts with the question of what is the minimal functionality we can deliver to the customer. This is the direction for the first 3 months. Once the MVP is out there, the product direction actually becomes agile. We take user feedback, strategic direction and market forces all into account o determine the next most important thing to deliver. It could be improving existing functionality, adding new features, responding to changing market conditions, any of the above. The product direction emerges. In fact, many of the things that we initially thought would be valuable when we started might never end up getting done because we would be agile with our product direction to produce that what our customers really want. We would not waste time building things we think our customers want and would instead quickly deliver and validate if we actually built the right thing. We would adapt and learn and even delete features if our customers don't use them. 

Prioritization

Our prioritization is now based on real use. We only work on the very limited scope that is to be delivered next. We have a sharp focus on the next most important customer request. Instead of everything being equally important and delivered at the same time, we are actually delivering software incrementally. This forces us to prioritize from the user's perspective. Instead of building a workflow that is seven levels deep, we wait for the business case for just enough workflow to become evident and then implement just what is needed by the customers. The number of assumptions in prioritization starts to dwindle to the assumptions only about the very next thing, and we validate or invalidate those assumptions as soon as often as we deliver.

The point about focus is worth repeating. You could probably get more value out the door with fewer people and in less time if the entire team is focussed on just the next deliverable as opposed to a single 3-year long deliverable. Alternate priority conversations don't ever interrupt current work. We don't pull the team off in the middle of solving business problems to estimate how long something else would take. When capacity is available, we prioritize just in time and pull the next most important thing. We don't spread ourselves thin and instead get focussed on the next thing to deliver.

Design

Analysis, design, and architecture all happen just in time as well. Architects act more as tech leads and enable the other developers on the team to achieve outcomes faster. The focus is on delivering the next valuable piece of software and not on building the perfect architecture. We architect knowing that we are making assumptions about what the next thing of value is. That means that while we require the flexibility to be able to change course, we also need the flexibility to delete entire pieces of the architecture without adversely affecting the customers. This leads to an architecture that is both emergent and flexible. It does not hinder today's progress and enables tomorrow's extension of functionality. We are wary of looking too far into the future because we don't know what we will need to build. If the current priority does not require it, we don't write it. We refactor, learn and adapt as time moves on, in order to make delivery of value as easy as possible. This is where the architects actually earn their technical swagger. Not in long-running pipe dream tech tasks, but in real value delivering functionality.

Developer Morale

Developers are focussed on producing the next valuable item. They get MVP feedback in three months as opposed to three years and continuous feedback after that. True agility exists when the distance between prioritization-code-delivery-feedback is minimized. Developers not only get feedback but also the sense of accomplishment of delivering customer value. Soon, they start gaining that sense of accomplishment based on solving business problems and not just delivering features since delivering features is a daily thing. 

Architects are a part of the team, and hence the technical decisions are pushed to the team level. Developers are in charge of their own destinies. they are not beholden to a central decision-making body that drops the gavel on what they can and cannot do. They have the ability to solve the problems in front of them the best way possible and have experienced architects on the team to consult. In summation, developers get to deliver value and get feedback, are in charge of their own technical destinies and get to solve the problems before them the best way possible. If any developer is unhappy in those surroundings, they probably need to look for an architect job at a company doing a three-year project.

(Short Term) Predictions

As opposed to figuring out what we can get done in three years and then making promises on those predictions, we move to a more tenable world. We start operating under a steady flow of value to the customer and make predictions on what we can get done in the next week or month. Sure, it makes long-term roadmap planning difficult, but that is for a good reason. Long term roadmap planning is based on a central assumption - We know everything that the customers want at the beginning of the project. I hope that in the modern day, no one believes that to be true for any decent length of time. Our ability to inspect and adapt in the short term is what is going to differentiate us from our customers. This ability gets severely undermined when we make long-term projections that bind us into promises and commitments that stop us from doing the most valuable things for our customers. We can be a lot more flexible and deliver the most valuable functionality as opposed to being locked into untenable promises.

Ending The Project

The product is at all times, after the MVP delivery, in a ready state. We are always delivering and validating what we assume to be the next highest value outcome. At some point though, what is the next highest value outcome for this product or project might not be the highest value outcome for the company. We might have also figured out, pretty early that this product was really not as valuable as we initially thought. Rather than sinking 3 years into this product we can actually end the project early and still have something that gives us return on investment. Instead of waiting to find out we have very early results on whether what we are working on is valuable or not. The very first assumption we make when we start work on a project or a product is that this has value. We need to find out if that is the case. Incremental and iterative delivery helps us find out as early as possible if that is true. It also provides us with multiple exit points from the project. We can exit at any point where we believe that any further investment in the product would be less valuable than investment somewhere else. This is in contrast with a "sunk cost " thinking in a 3-year project where there is only one easy exit point, at the end of the three years.

Also...

The Value Proposition

There is the small matter of making money. Imagine if delivering one-sixth of the product at 6 months can help you charge one-sixth of the projected price. At every six month increment, you can up the price by one-sixth of the projected price, till you are charging the same price as you expected at the end of the three-year term. This billing scheme would mean that by the time you end the fourth year of the project, you would have made almost twice the amount of money with the iterative scheme. Of course, there are assumptions here that this billing scheme is possible. There probably wont be as many customers signing on to the earlier versions. The billing scheme could be better than the one proposed here to make up for the fewer customer sign ups early on. More than the early and added return on investment, we actually find out very early if we are building something that people would be willing to pay for.

But My Project Is "Different"

None of this is to say that there are no successful three-year projects. I have been involved in multi-year projects before. Even in those, we went to production halfway through the proposed timeline with half the functionality and continued to release every day. We included customer feedback in prioritization. If I were to do the project again, I would have done it slightly differently and gone to production with real customers much earlier. As early as 10% of the functionality being done, that is probably the point where our MVP was. We were lucky though, the traditional organization elements of architecture and project management left us alone for the most part. It helped, but going to market early would probably have helped even more. Long running projects can be successful, but I am willing to bet that the same project, when done iteratively, and brought truly to production earlier would be a lot more on the mark and truly deliver the benefits of agile development.

In my estimation (I don't have hard proof on this), about 98% of the projects can be done in this iterative manner. Unless you are putting a man in space and have no way to user test it, this approach of going to market with 10% of the product, would benefit greatly. It could help even an otherwise successful project be even more successful. If you, the person reading this, believe that your project falls in the 2% case, there is a pretty good chance that you are mistaken. Most likely, you can define the MVP for your project, release it and then grow it. Yes, there are reasons to do it the other way, sure, there are people to convince, of course, it is a hard sell, but I am hoping we can agree that it is worth a try.

...But We Have Agile Teams

Yes, there is room for agile teams in long running projects. Unfortunately, most of the time, this is just teams running really fast in what is assumed to be the right direction. This assumption is not validated till late. Also, in a traditional organizational structure, with multiple levels of management, traditional PMO, top-heavy architecture org, expert consultants and zero contact between developers and end users, these teams are nothing but sports cars in a traffic jam. Yes, they can go really fast, but they cannot move anywhere. The core tenant of Agile - Inspect and Adapt, does not exist at the most important level. We are not actually taking the product to market and inspecting the results and changing course. We are instead relying on heavy up-front analysis and design. In complex systems, iteration would beat analysis any day.

The funny part is that these teams are set up to fail in the context of a system that is anything but agile. They are asked to be agile, to produce products fast and with high quality. Meanwhile, they live in the center of a system, where results and timelines are pre-determined. The system is set up to not validate assumptions. The teams might be agile, but the system being traditional undermines that agility. Attempts are made to make the team "more" agile and to fix their processes. That, in itself, is a waste of the company’s resources, as the real agility is needed at the top levels of management. An agile management team would allow the entire system to be optimized for early and frequent delivery.

Finally

If you are in the middle of a long project that is scheduled for a big bang release, there is still time to pivot. It will be a little messy, but in my opinion completely worth it. You will have to figure out what the MVP would look like. Make hard choices about which current features in progress to throw away. Maybe even reduce the size of the team in order to get focused on what is needed for the MVP. The good news is you have a start already and potentially are at a point where with some finishing touches, you can actually put together an MVP. Once that is in its place, it is a matter of delivering value as quickly as possible. Yes, if the company does not have a safe culture, this could be politically hazardous to attempt. After all, in the absence of safety, influencing change and doing the right thing is often a personal and a political risk.

There are no individuals that intend for a project to be a failure. The system, in the case of a big bang project, is set up for failure (or "less success"). Individuals are doing their best to make things happen based on what they are asked to do. Regardless of whether they are developers, architects, management or project managers, they all are working hard towards success. The system though is set up to be in their way. It is in the best interest of not just the company but the employees for us to switch to a more iterative approach.

Also, in this post, 3 years and 3 months are just an example. This is context specific. In your context, a long-running project could be 6 months and MVP be 2 weeks. Anything with no delivery for over a year though, in most cases, should be considered a long-running project.