Promises, promises

What is it about the waterfall model of software development? Why do we use this model?

A little background: The waterfall model has been used in software development for the past forty years, almost as long as we have had software development itself. It consists of breaking a development effort into multiple steps, usually 'analyze', 'design', 'code', 'test', and 'deploy'. (Some shops will have different names for these steps, and some shops will add or remove steps.)

But the waterfall model is accepted as deficient. The seminal paper on the subject, written by Winston Royce, lists this method as possible but he advises against it. (It didn't work in the early days of software development and it works poorly today. The Standish group has a number of studies that show waterfall-run projects are consistently over-budget, late, and result in low quality software.)

Yet we use the waterfall model. And continue to use it, despite the development of other techniques. The current contender is called 'agile' and it works very differently from waterfall.

More background: Agile development discards the notion of 'big design up front' and distinct steps for design, coding, and testing. Instead, it relies on small iterations of deciding on key features, developing automated tests for those features, and then coding the features. The users (or stakeholders) get to decide which features go in first. The development team works on the features until they are complete and then you begin another iteration. You repeat iterations until you have all features you want, and then you ship the product.

The curious reader might be asking "But when do you test? I see no test phase!" Agile development does not require, in fact avoids, a big test effort at the end of the development effort. It puts testing in each iteration, and requires automated testing on each iteration. The 'automated' adjective for testing is important. It allows everyone on the team to run tests, not just a set of test/QA folks. Automated testing also allows *all* tests to be run, so you know that everything you have done to date works. Iteration 2 runs all of iteration 2 tests and all of iteration 1 tests. If a change for iteration 2 breaks something coded in iteration 1, you know about it immediately, and you fix it before proceeding with iteration 3. This discipline is required for the promise that agile development makes.

More on this promise later. For now, I will ramble on about the reasons we keep using waterfall methods.

Some believe that project managers use waterfall because they learned it when they were in school and they stick with it. (Equivalent to 'you cannot teach an old dog new tricks.) Some believe that project managers are capable of learning a new method but see no benefit and are unwilling to risk their career on something untried. Some believe that in large organizations, a sort of 'lock-in' occurs, where all managers use waterfall, expect all other managers to use waterfall, and build there plans around waterfall. When a 'maverick' manager starts using a different process, his plans don't fit in well with the rest of the team, and the rest of the team persuades him to convert his plan (and therefore his process) back to the norm. In this case, the organization has learned waterfall and is unable to learn a new trick.

I believe the reason managers (and organizations) stay with waterfall project plans is because of the promise that a waterfall schedule makes: It promises to deliver X functionality on date Y. (Where X is some random set of agreed-upon features and Y is some agreed-upon date.) This is a very seductive promise; it provides certainty and can be used to 'hold people accountable'.

Agile development methods make a very different promise. They do not promise specific features on a specific date. They promise that the product will always be shippable and that the most important features will be coded first. This promise does not provide the certainty that a set of features will be available on a specific date, which is a very different promise.

Waterfall and Agile address the 'Iron Triangle' (speed, cost, quality -- pick any two) very differently. Waterfall defines speed and cost (and attempts to control quality); agile defines cost and quality (and measures speed but makes no attempt to control it).

Two very different approaches to project planning.