Agile is for startups; waterfall for established projects

The Agile method was conceived as a revolt against the Waterfall method. Agile was going to be everything that Waterfall was not: lightweight, simple, free of bureaucracy, and successful. In retrospect, Agile was successful, but that doesn't mean that Waterfall is obsolete.

It turns out that Agile is good for some situations, and Waterfall is good for others.

Agile is effective when the functionality of the completed system is not known in advance. The Agile method moves forward in small steps which explore functionality, with reviews after each step. The Agile method also allows for changes in direction after each review. Agile provides flexibility.

Waterfall is effective when the functionality and the delivery date is known in advance. The Waterfall method starts with a set of requirements and executes a plan for analysis, design, coding, testing, and deployment. It commits to delivering that functionality on the delivery date, and does not allow (in its pure form) for changes in direction. Waterfall provides predictability.

Established companies, which have an ongoing business with procedures and policies, often want the predictability that Waterfall offers. They have business partners and customers and people to whom they make commitments (such as "a new feature will be ready for the new season"). They know in detail the change they want and they know precisely when they want the change to be effective. For them, the Waterfall method is appropriate.

Start-ups, on the other hand, are building their business model and are not always certain of how it will work. Many start-ups adjust their initial vision to obtain profitability. Start-ups don't have customers and business partners, or at least not with long-standing relationships and specific expectations. Start-ups expect to make changes to their plan. While they want to start offering services and products quickly (to generate income) they don't have a specific date in mind (other than the "end of runway" date when they run out of cash). Their world is very different from the world of the established company. They need the flexibility of Agile.

The rise of Agile did not mean the death of Waterfall -- despite some intentions of the instigators. Both have strengths, and each can be used effectively. (Of course, each can be mis-used, too. It is quite easy to manage a project "into the ground" with the wrong method, or a poorly applied method.)

The moral is: know what is best for your project and use the right methods.

Small check-ins saved me

With all of the new technology, from cloud computing to tablets to big data, we can forget the old techniques that help us.

This week, I was helped by one of those simple techniques. The technique that helped was frequent, small check-ins to version control systems. I was using Microsoft's TFS, but this technique works with any system: TFS, Subversion, git, CVS, ... even SourceSafe!

Small, frequent changes are easier to review and easier to revert than large changes. Any version control system accepts small changes; the decision to make large or small changes is up to the developer.

After a number of changes, the team with whom I work discovered a defect, one that had escaped our tests. We knew that it was caused by a recent change -- we tested releases and found that it occurred only in the most recent release. That information limited the introduction of the defect to the most recent forty check-ins.

Forty check-ins may seem like a long list, but we quickly identified the specific check-in by using a binary search technique: get the source from the middle revision; if the error occurs move to the earlier half, if not move to the later half and start in that half's middle.

The real benefit occurred when we found the specific check-in. Since all check-ins were small, this check-in was too. (It was a change of five different lines.) It was easy to review the five individual lines and find the error.

Once we found the error, it was easy to make the correction to the latest version of the code, run our tests (which now included an addition test for the specific problem we found), verify that the fix was correct, and continue our development.

A large check-in would have required much more examination, and more time.

Small check-ins cost little and provide easy verification. Why not use them?

Refactoring is almost accepted as necessary

The Agile Development process brought several new ideas to the practice of software development. The most interesting, I think, is the notion of re-factoring as an expected activity.

Refactoring is the re-organization of code, making it more readable and eliminating redundancies. It is an activity that serves the development team; it does not directly contribute to the finished product.

Earlier methods of software development did not list refactoring as an activity. They made the assumption that once written, the software was of sufficient quality to deliver. (Except for defects which would be detected and corrected in a "test" or "acceptance" phase.)

Agile Development, in accepting refactoring, allows for (and encourages) improvements to the code without changing the behavior of the code (that is, refactoring). It is a humbler approach, one that assumes that members of the development team will learn about the code as the write the code and identify improvements.

This is a powerful concept, and, I believe, a necessary one. Too many projects suffer from poor code quality -- the "technical backlog" or "technical debt" that many developers will mention. The poor code organization slows development, as programmers must cope with fragile and opaque code. Refactoring improves code resilience and improves visibility of important concepts. Refactored code is easier to understand and easier to change, which reduces the development time for future projects.

I suspect that all future development methods will include refactoring as a task. Agile Development, as good as it is, is not the perfect method for all projects. It is suited to projects that are exploratory in nature, projects that do not have a specific delivery date for a specific set of features.

Our next big software development method may be a derivative of Waterfall.

Agile Development (and its predecessor, Extreme Programming) was, in many ways, a rebellion against the bureaucracy and inflexibility of Waterfall. Small teams, especially in start-up companies, adopted it and were rewarded. Now, the larger, established, bureaucratic organizations envy that success. They think that adopting Agile methods will help, but I have yet to see a large organization successfully merge Agile practices into their existing processes. (And large, established, bureaucratic organizations will not convert from Waterfall to pure Agile -- at least not for any length of time.)

Instead of replacing Waterfall with Agile, large organizations will replace Waterfall with an improved version of Waterfall, one that keeps the promise of a specific deliverable on a specific date yet adds other features (one of which will be refactoring). I'm not sure who will develop it; the new process (let's give it a name and call it "Alert Waterfall") may come from IBM or Microsoft or Amazon.com, or some other technical leader.

Yet it will include the notion of refactoring, and with it the implicit declaration that code quality is important -- that it has value. And that will be a victory for developers and managers everywhere.

Agile is not for everyone

Agile development is a hot topic for many project managers. It is the latest management fad, following total cost of ownership (TCO) and total quality management (TQM). (Remember those?)

So here is a little bit of heresy: Agile development methods are not for every project.

To use agile methods effectively, one must understand what they offer -- and what they don't.

Agile methods make one guarantee: that your system will always work, for the functionality that has been built. Agile methods (stakeholder involvement, short development cycles, automated testing) ensures that the features you build will work as you expect. Even as you add new features, your automated tests ensure that old features still work. Thus, you can always send the most recent version of your system to your customers.

But agile methods don't make very promise. Specifically, they don't promise that all of your desired features will be available (and working) on a specific date. You may have a list of one hundred features; agile lets you implement those features in a desired order but does not guarantee that they will all be completed by an arbitrary date. The guarantee is only that the features implemented by that date will work. (And you cannot demand that all features be implemented by that date -- that's not agile development.)

Agile does let you make projections about progress, once you have experience with a team, the technology, and a set of features for a system. But these projections must be based on experience, not on "gut feel". Also, the projections are just that: projections. They are estimates and not guarantees.

Certain businesses want to commit to specific features on specific dates, perhaps to deliver a system to a customer. If that is your business, that you should look carefully at agile methods and understand what they can provide. It may be that the older "waterfall" methods, which do promise a specific set of features on a specific date, are a better match.

Waterfall caused specialization; agile causes generalization

There are a number of differences between waterfall processes and agile processes. Waterfall defines one long, segmented process; agile uses a series of short iterations. Waterfall specifies a product on a specific date; agile guarantees a shippable product throughout the development process.

Another difference between waterfall and agile is the specialization of participants. Waterfall projects are divided into phases: analysis, development, testing, etc. and these phases tend to be long themselves. Agile projects have the same activities (analysis, development, testing) but on a much shorter timeframe. A waterfall project may extend for six months, or a year, or several years, and the phases of those projects may extend for months -- or possibly years.

The segmentation of a waterfall project leads to specialization of the participants. It is common to find a waterfall project staffed by analysts, developers, and testers, each a distinct team with distinct management teams. The different teams use tools specific to their tasks: databases for requirements, compilers and integrated development environments, test automation software and test case management systems.

This specialization is possible due to the long phases of waterfall projects. It is reasonable to have team "A" work on the requirements for a project and then (when the requirements are deemed complete) assign team "B" to the development phase. While team "B" develops the project, team "A" can compose the requirements for another project. Specialization allows for efficient scheduling of personnel.

Agile processes, in contrast, have short iterations of analysis, design, coding, and testing. Instead of months (or years), an iteration may be one or two weeks. With such a short period of time, it makes little sense to have multiple teams work on different aspects. The transfer of knowledge from one team to another, a small task in a multi-year project, is a large effort on a two-week iteration. Such inefficiencies are not practical for short projects, and the better approach is for a single team to perform all of the tasks. (Also, the two-week iteration is not divided into a neat linear sequence of analysis, design, development, and test. All activities occur throughout the iteration, with multiple instances of each task.)

A successful agile process needs people who can perform all of the tasks. It needs not specialists but generalists.

Years of waterfall projects have trained people and companies into thinking that specialists are more efficient than generalists. (There may be a bit of Taylorism here, too.) Such thinking is so pervasive that one finds specialization in the company's job descriptions. One can find specific job descriptions for business analysts, developers, and testers (or "QA Analysts").

The shift to agile projects will lead to a re-thinking of specialization. Generalists will be desired; specialists will find it difficult to fit in. Eventually, generalists will become the norm and specialists will become, well, special. Even the job descriptions will change, with the dominant roles being "development team members" with skills in all areas and a few specialist roles for special tasks.

Unstoppable cannon balls and immovable posts

There is an old riddle: what happens when an unstoppable cannon ball strikes an immovable post?

The unstoppable cannon ball is a hypothetical construct. It is a cannon ball that, once in motion, stays in motion. (It's ability to defy friction comes from its hypothetical nature.)

The immovable post is also a hypothetical construct. It is a post that, once placed, cannot be moved.

While the riddle is entertaining, a real-world version of it is emerging with software development methodologies. Instead of an unstoppable cannon ball and an immovable post we have the waterfall methodology and the agile methodology. Perhaps the agile method is the equivalent of the cannon ball, as both emphasize motion. It doesn't really matter, though.

What matters is that the two methodologies are incompatible. One can develop software using one methodology, or the other, but one cannot use both. For small shops that have only one project (say a start-up), this is not an issue. Larger shops (any shop with multiple projects) must choose a methodology; you cannot use waterfall for some projects and agile methods for another. (Well, you can, provided that the two sets of projects have no interaction. But that is unlikely.)

The waterfall and the agile methods make two different promises. Waterfall promises a specific deliverable, with a specific set of features, and a specific quality, on a specific date. Agile promises that the product is always shippable (that is, very high quality), and promises to implement the most important features first, but makes no promises about a complete set of features on a specific date.

It is easy to dismiss these differences if we think of these approaches as methods of developing software. It is harder to dismiss them when we realize the truth: waterfall and agile are not methods for building software but philosophies for managing projects.

Managing projects is serious business, and is the core of most IT shops. (Projects can by managed by non-IT shops, too, but I will stick with the IT world.)

One cannot run a business (a successful business) with multiple coordinated projects that use project practices which differ to the extent that waterfall and agile differ. Waterfall, with its "big design up front" and locked-in delivery dates cannot mesh with agile, with its "choose as you go" method for selecting features. The rise of agile methods -- agile project management -- means that companies must choose one or the other. Which one is important, too, and depends on how the managers of the company want to run their company.

This schism has far-reaching effects. It can be felt in the acquisition of other companies, when one merges the management of project teams. It affects the hiring of new employees: are they familiar with the practices of the hiring company? It can even affect joint projects between companies.

I don't claim that one is better than the other. My preference is for agile project management, based on my experience and projects. It may be the right methodology for you. Or waterfall may be the better methodology for you. Managers must evaluate both and consider the benefits and costs.

My advice is to learn about both and make an informed decision.

The Agile and Waterfall Impedance Mismatch

During a lunchtime conversation with a colleague, we chatted about project management and gained some insights into the Waterfall and Agile project management techniques.

Our first insight was that an organization cannot split its project management methods. It cannot run some projects with a Waterfall process and other projects with Agile processes. (At least not if the projects must coordinate deliverables.) A company that uses Waterfall processes may be tempted to run a pilot project with Agile processes -- but if that pilot project "plugs in" to other projects, the result will be failure.

The problem is that Waterfall and Agile make two different promises. Waterfall promises a specific set of functionality, with a specific level of quality, delivered on a specific date. Agile makes no such promise;  it promises to add functionality and have a product that is always ready to ship (that is a high level of quality), albeit with an unknown set of functionality. The Agile process adds small bits of functionality and waits to get them correct before adding others -- thus ensuring that everything that has been added is working as expected, but not promising to delivery everything desired by a specific date. (I am simplifying things here. Agile enthusiasts will point out that there is quite a bit more to Agile processes.)

Waterfall processes make promises that are very specific in terms of feature set, quality, and delivery time -- but are not that good at keeping them. Hence, we have a large number of projects that are late or have low quality. Agile makes promises that are specific in terms of quality, and are good at keeping them. But the promises of the Agile processes are limited to quality; they do not propose the specifics that are promised by Waterfall.

With two different promises, it is no surprise that waterfall and agile processes don't co-exist. (There are other reasons that the two methods fail to cooperate, including the "design up front" of Waterfall and the "design as you go" of Agile.)

Our second insight was that transitioning an IT shop from Waterfall to Agile methods should not be accomplished by pilot projects.

Pilot projects are suitable to introduce people to the methods -- but those pilot projects must exist in isolation from the "regular" projects. Such projects were easy to establish in the earlier age of separate systems -- "islands of automation" -- that gave each system a measure of independence. Today, it is rare to see an IT system that exists in isolation.

Rather that use pilot projects, we like the idea of introducing ideas from Agile into the standard process used within the company. Our first choice is automated testing, the unit-level automated testing that can be performed by developers. Giving each developer the ability to run automated tests introduces them to a core practice of Agile, without creating an impedance mismatch.

After automated testing, we like the idea of allowing refactoring. Often omitted from plans for Waterfall projects, refactoring is another key practice of Agile development. Once unit tests are in place, developers can refactor with confidence.

Our third insight relates to project methods and project size. We think (and this is speculation) that Agile is better suited to small projects (and small systems) and Waterfall may be better suited to large systems. Thus, if you have large, complex systems, you may want to stay with Waterfall; if you have small systems (or even small applications) then you may want to use Agile.

We think that this relationship is correlation, not causal. That is, you can pick one side of the equation and drive the other. If you have large systems, you will end up with Waterfall. But the equation works in both directions, and you don't have to start with the size of your systems. If you use Agile methods, you will end up with smaller, collaborating systems.

Now, we realize that a lot of companies have large systems. We don't believe that by switching overnight to Agile methods will lead to smaller systems overnight. A base of large systems contains a certain amount of inertia, and requires a certain effect to redesign into smaller systems.

What we do believe is that you have a choice, that you can choose to use Agile (or Waterfall) methods (carefully, avoiding impedance mismatches), and that you can change the size of your systems. You can manage the size and complexity of your systems by selecting your processes.

These are choices that managers should accept and make carefully. They should not assume that "the system is the system" and "things are big and complicated and will always be so".