Agile, Waterfall, and Risk

For some years (decades, really), software development has used an agile approach to project management. The Agile method sees short iterations that each focus on a single feature, with the entire team reviewing progress and selecting the feature for the next iteration. Over time, a complete system evolves. The advantage is that the entire team (programmers, managers, salespersons, etc.) learn about the business problem, the functions of the system, and the capabilities of the team. The team can change course (hence the name "agile") as they develop each feature.

Prior to Agile, for some years (decades, really), software development used the "waterfall" approach to project management. The Waterfall method starts with a set of requirements and a schedule, and moves through different phases for analysis, design, coding, testing, and deployment. The important aspect is the schedule. The Waterfall method promises to deliver a complete system on the specified date.

This last aspect of Waterfall is quite different from Agile. The Agile method makes no promise to deliver a completed system on a specific date. It does promise that each iteration ends with a working system that implements the features selected by the team. Thus, a system developed with Agile is always working -- although incomplete -- whereas a system developed with Waterfall is not guaranteed to work until the delivery date.

(It has been observed that while the Waterfall method promises a complete, working system on the specified delivery date, it is quite poor at keeping that promise. Many projects overrun both schedule and budget.)

Here is where risk comes into play.

With Agile, the risk is shared by the entire team, key among these are developers and managers. An agile project has no specified delivery date, but more often than not senior managers (those above the agile-involved managers) have a date in mind. (And probably a budget, too.) Agile projects can easily overrun these unstated expectations. When they do, the agile-involved managers are part of the group held responsible for the failure. Managers have some risk.

But look at the Waterfall project. When a waterfall project fails (that is, runs over schedule or budget) the managers have way to distance themselves from the failure. They can say (honestly) that they provided the developers with a list of requirements and a schedule (and a budget) and that the developers failed to meet meet the "contract" of the waterfall project. Managers can deflect the risk to the development team.

(For some reason, we rarely question the feasibility of the schedule, or the consistency and completeness of the requirements, or the budget assigned to the project. These are considered "good", and any delay or shortcoming is therefore the fault of the developers.)

Managers want to avoid risk -- or at least transfer it to another group. Therefore, I predict that in the commercial space, projects will slowly revert from Agile methods to Waterfall methods.

The fall and possible rise of UML

Lots of folks are discussing UML, and specifically the death of UML. What killed UML? Lots of people have different ideas. I have some ideas too. Rather than pin the failure on one reason, I have a bunch.

First, our methods changed, and UML was not a good fit with newer methods. UML was created in the world of large-scale waterfall projects. It works well with those projects, with design up front (disparagingly called "Big Design Up Front") as a precursor to coding. UML does not work well with Agile methods, where design and coding occur in parallel. UML assigns value to code; the idea of up-front design is to build the right code form the start and not revise it. In the UML world, changes to code are expensive and to be avoided. UML also attaches value to the designs, with identical attachments to designs and the desire to avoid changes to designs. (Although changes to designs are preferred over changes to code.)

UML works well with object-oriented programming, but not with cloud computing (small scripts instead of big code).

Second, UML entailed costs. UML notation was difficult to learn. Or at least required some time to learn. The tools took time to learn, and they also cost significant sums. The mindset was "invest now (by learning UML and buying the tools) to prevent more costly mistakes later". At the time, there were charts showing the cost of a mistake, and comparing the cost of detecting the mistake at different points in the project. A mistake detected early (say in requirements or design) was less expensive than a mistake detected later (say in coding or testing). Mistakes detected after deployment were the most expensive. This effect justified the expense of UML tools.

But UML tools were expensive, and not everyone on the team got UML tools. The tools were reserved for the designers; coders were limited to printed copies of UML diagrams. This lead to the notion that designers were special and worth more than programmers. (The elite received UML tools; the plebes did not.) This in turn lead to resentment by programmers.

A third (and often overlooked) reason was the expense for designers. When programmers performed both design and programming, their salaries covered both activities. UML formalized the design process and required a subteam of designers, and each of those designers required a salary. (And they often wanted salaries higher than those of programmers.)

A fourth (and also often overlooked) reason was the added delay to the development process.

UML created an additional step in the waterfall process. Theoretically, it did not, because UML was simply formalized design documents. But in practice, UML did create an additional step.

Before UML, a project would have the formal steps of requirements, design, coding, testing, and deployment. That's what managers thought they had. In reality, the steps were slightly different than those formal steps. The actual steps were requirements, design and coding, testing, and deployment.

Before UML: requirements -> design and code -> test -> deploy

Notice that the steps of design and code are one step, not two. It was an activity performed by the programming team. As it was a single team, people could move from designing to coding and back again, revising the design as they developed the code.

UML and a formal design deliverable changed the process to the five steps the managers thought they had:

With UML: requirements -> design -> code -> rest -> deploy

UML forced the separation of design from coding, and in doing so, changed the (informal) four-step process to a five-step process.

Programmers were used to designing as well as programming. With UML, programmers could not unilaterally change the design; they had to push back against the design. This set up conflicts between designers and programmers. Sometimes the designers "gave in" and allowed a change; other times they "held fast" and programmers had to build something they considered wrong. In either case, such differences introduced delays and political struggles when there were none before.

Those are my observations for UML, and why it failed: new methods not suitable for UML, direct expense of tools and training, direct expense of designers, and a slower development process.

* * * * *

In a way, I am sorry for the loss of UML. I think it can be a helpful tool. But not in the way it was implemented.

UML was added to a project as a design aid, and one that occurred prior to coding. Perhaps it is better to have UML as a diagnostic instead of an aspiration. That is, instead of creating UML and then generating code from UML, create code and then generate UML from the code.

In this way, UML could be a kind of "super lint" that reports on the design of the system.

There was "round-tripping" which allowed for UML to be converted to code, and then that code converted back to UML. That is not the same; it leaves UML as the center for design. And round-tripping never really worked the way we needed. A one-way code-to-UML diagnostic puts code at the center and UML as a tool to assist the programmers. (That's my bias as a programmer showing.)

A code-to-UML diagnostic could be helpful to Agile projects, just as 'lint' and other style checkers are. The tools may be less expensive (we've gotten better at tools, and a diagnostic tool is easier to build than a UML editor). We would not have a separate design team, avoiding that expense (and the associated politics). And a diagnostic tool would not slow the development process -- or at least not so much.

Maybe we will see such a tool. If we do, it will have to be developed by the open-source community. (That is, an individual who wants to scratch an itch, much like Perl, or Python, or Linux.) I don't see a large corporation building one; I don't see a business model for it.

Anyone want to scratch an itch?

What Agile and Waterfall have in common

Agile and Waterfall are often described in contrasts: Waterfall is large and bureaucratic, Agile is light and nimble. Waterfall is old, Agile is new. Waterfall is... you get the idea.

But Waterfall and Agile have things in common.

First and most obvious, Waterfall and Agile are both used to manage development projects. They both deliver software.

I'm sure that they are both used to deliver things other than software. They are tools for managing projects, not limited to software projects.

But those are the obvious common elements.

An overlooked commonality is the task of defining small project steps. For Waterfall, this is the design phase, in which requirements are translated into system design. The complete set of requirements can paint a broad picture of the system, providing a general shape and contours. (Individual requirements can be quite specific, with details on input data, calculations, and output data.)

Breaking down the large idea of the system into smaller, code-able pieces is a talent required for Waterfall. It is how you move from requirements to coding.

Agile also needs that talent. In contrast to Waterfall, Agile does not envision the completed system and does not break that large picture into smaller segments. Instead, Agile asks the team to start with a small piece of the system (most often a core function) and build that single piece.

This focus on a single task is, essentially, the same as the design phase in Waterfall. It converts a requirement (or user story, or use case, or whatever small unit is convenient) into design for code.

The difference between Waterfall and Agile is obvious: Waterfall converts all requirements in one large batch before any coding is started, and Agile performs the conversions serially, seeing one requirement all the way to coding and testing (or more properly, testing and then coding!) before starting the next.

So whether you use Waterfall or Agile, you need the ability to "zoom in" from requirements to design, and then on to tests and code. Waterfall and Agile are different, but the differences are more in the sequence of performing tasks and not the tasks themselves.

Agile can be cheap; Waterfall is expensive

Agile can be cheap, but Waterfall will always be expensive.

Here's why:

Waterfall starts its process with an estimate. The Waterfall method uses a set of phases (analysis, design, coding, testing, and deployment) which are executed according to a fixed schedule. Many Waterfall projects assign specific times to each phase. Waterfall needs this planning because it makes a promise: deliver a set of features on a specific date.

But notice that Waterfall begins with an estimate: the features that can be implemented in a specific time frame. That estimate is crucial to the success of the project. What is necessary to obtain that estimate?

Only people with knowledge and experience can provide a meaningful estimate. (One could, foolishly, ask an inexperienced person for the estimate, but that estimate has no value.)

What knowledge does that experienced person need? Here are some ideas:

- The existing code
- The programming language and tools used
- The different teams involved in development and testing
- The procedures and techniques used to coordinate efforts
- The terms and concepts used by the business

With knowledge of these, a person can provide a reasonable estimate for the effort.

These areas of knowledge do not come easily. They can be learned only by working on the project and in different capacities.

In other words, the estimate must be provided by a senior member of the team.

In other words, the team must have at least one senior member.

Waterfall relies on team members having knowledge about the business, the code, and the development processes.

Agile, in contrast, does not rely on that experience. Agile is designed to allow inexperienced people work on the project.

Thus, Agile projects can get by without senior, experienced team members, but Waterfall projects must have at least one (and probably more) senior team members. Since senior personnel are more expensive than junior, and Waterfall requires senior personnel, we can see that Waterfall projects will, on average, cost more than Agile projects. (At least in terms of per-person costs.)

Do not take this to mean that you should run all projects with Agile methods. Waterfall may be more expensive, but it provides different value. It promises a specific set of functionality on a specific date, a promise that Agile does not make. If you need the promises of Waterfall, it may be worth the extra cost (higher wages). This is a business decision, similar to using proprietary tools over open-source tools, or leasing premium office space in the suburbs over discount office space in a not-so-nice part of town.

Which method you choose is up to you. But be aware that they are not the same, not only in terms of deliverables but in staffing requirements. Keep those differences in mind when you make your decision.

Agile is for startups; waterfall for established projects

The Agile method was conceived as a revolt against the Waterfall method. Agile was going to be everything that Waterfall was not: lightweight, simple, free of bureaucracy, and successful. In retrospect, Agile was successful, but that doesn't mean that Waterfall is obsolete.

It turns out that Agile is good for some situations, and Waterfall is good for others.

Agile is effective when the functionality of the completed system is not known in advance. The Agile method moves forward in small steps which explore functionality, with reviews after each step. The Agile method also allows for changes in direction after each review. Agile provides flexibility.

Waterfall is effective when the functionality and the delivery date is known in advance. The Waterfall method starts with a set of requirements and executes a plan for analysis, design, coding, testing, and deployment. It commits to delivering that functionality on the delivery date, and does not allow (in its pure form) for changes in direction. Waterfall provides predictability.

Established companies, which have an ongoing business with procedures and policies, often want the predictability that Waterfall offers. They have business partners and customers and people to whom they make commitments (such as "a new feature will be ready for the new season"). They know in detail the change they want and they know precisely when they want the change to be effective. For them, the Waterfall method is appropriate.

Start-ups, on the other hand, are building their business model and are not always certain of how it will work. Many start-ups adjust their initial vision to obtain profitability. Start-ups don't have customers and business partners, or at least not with long-standing relationships and specific expectations. Start-ups expect to make changes to their plan. While they want to start offering services and products quickly (to generate income) they don't have a specific date in mind (other than the "end of runway" date when they run out of cash). Their world is very different from the world of the established company. They need the flexibility of Agile.

The rise of Agile did not mean the death of Waterfall -- despite some intentions of the instigators. Both have strengths, and each can be used effectively. (Of course, each can be mis-used, too. It is quite easy to manage a project "into the ground" with the wrong method, or a poorly applied method.)

The moral is: know what is best for your project and use the right methods.

Agile values clean code; waterfall may but doesn't have to

Agile and Waterfall are different in a number of ways.

Agile promises that your code is always ready to ship. Waterfall promises that the code will be ready on a specific date in the future.

Agile promises that your system passes the tests (at least the tests for code that has been implemented). Waterfall promises that every requested feature will be implemented.

There is another difference between Agile and Waterfall. Agile values clean code; Waterfall values code that performs as intended but has no notion of code quality. The Agile cycle includes a step for refactoring, a time for developers to modify the code and improve its design. The Waterfall method has no corresponding step or phase.

Which is not to say that Waterfall projects always result in poorly designed code. It is possible to build well-designed code with Waterfall. Agile explicitly recognizes the value of clean code and allocates time for correcting design errors. Waterfall, in contrast, has its multiple phases (analysis, design, coding, testing, and deployment) with the assumption that working code is clean code -- or code of acceptable quality.

I have seen (and participated in) a number of Waterfall projects, and the prevailing attitude is that code improvements can always be made later, "as time allows". The problem is that time never allows.

Many project managers have the mindset that developers should be working on features with "business value". Typically these changes fall into one of three categories: feature to increase revenue, features to reduce costs, and defect corrections. The mindset also considers any effort outside of those areas to be not adding value to the business and therefore not worthy of attention.

Improving code quality is an investment in the future. It is positioning the code to handle changes -- in requirements or staff or technology -- and reducing the effort and cost of those changes. In this light, Agile is looking to the future, and waterfall is looking to the past (or perhaps only the current release).

Refactoring is almost accepted as necessary

The Agile Development process brought several new ideas to the practice of software development. The most interesting, I think, is the notion of re-factoring as an expected activity.

Refactoring is the re-organization of code, making it more readable and eliminating redundancies. It is an activity that serves the development team; it does not directly contribute to the finished product.

Earlier methods of software development did not list refactoring as an activity. They made the assumption that once written, the software was of sufficient quality to deliver. (Except for defects which would be detected and corrected in a "test" or "acceptance" phase.)

Agile Development, in accepting refactoring, allows for (and encourages) improvements to the code without changing the behavior of the code (that is, refactoring). It is a humbler approach, one that assumes that members of the development team will learn about the code as the write the code and identify improvements.

This is a powerful concept, and, I believe, a necessary one. Too many projects suffer from poor code quality -- the "technical backlog" or "technical debt" that many developers will mention. The poor code organization slows development, as programmers must cope with fragile and opaque code. Refactoring improves code resilience and improves visibility of important concepts. Refactored code is easier to understand and easier to change, which reduces the development time for future projects.

I suspect that all future development methods will include refactoring as a task. Agile Development, as good as it is, is not the perfect method for all projects. It is suited to projects that are exploratory in nature, projects that do not have a specific delivery date for a specific set of features.

Our next big software development method may be a derivative of Waterfall.

Agile Development (and its predecessor, Extreme Programming) was, in many ways, a rebellion against the bureaucracy and inflexibility of Waterfall. Small teams, especially in start-up companies, adopted it and were rewarded. Now, the larger, established, bureaucratic organizations envy that success. They think that adopting Agile methods will help, but I have yet to see a large organization successfully merge Agile practices into their existing processes. (And large, established, bureaucratic organizations will not convert from Waterfall to pure Agile -- at least not for any length of time.)

Instead of replacing Waterfall with Agile, large organizations will replace Waterfall with an improved version of Waterfall, one that keeps the promise of a specific deliverable on a specific date yet adds other features (one of which will be refactoring). I'm not sure who will develop it; the new process (let's give it a name and call it "Alert Waterfall") may come from IBM or Microsoft or Amazon.com, or some other technical leader.

Yet it will include the notion of refactoring, and with it the implicit declaration that code quality is important -- that it has value. And that will be a victory for developers and managers everywhere.

Waterfall or Agile -- or both

A lot has been written (and argued) about waterfall methods and agile methods. Each has advocates. Each has detractors. I take a different path: they are two techniques for managing projects, and you may want to use both.

Waterfall, the traditional method of analysis, design, coding, testing, and deployment, makes the promise of a specific deliverable at a specific time (and at a specific cost). Agile, the "young upstart" makes the promise of frequent deliverables that function -- although only with what has been coded, not necessarily everything you may want.

Waterfall and agile operate in different ways and work in different situations. Waterfall works well when you and your team know the technology, the tools, the existing code, and the business rules. Agile works when you and your team are exploring new areas (technology, code, business rules, or a combination). Agile provides the flexibility to change direction quickly, where waterfall locks you in to a plan.

Waterfall does not work when there are unknowns. A new technology, for example. Or a new team looking at an existing code base. Or perhaps significant changes to business rules (where "significant" is smaller than you think.) Waterfall's approach to define everything up front cannot handle the uncertainties, and its schedules are likely to fail.

If your shop has been building web applications and you decide to switch to mobile apps, you have a lot of uncertainties. New technologies, new designs for applications, and changes to existing web services are required. You may be unable to list all of the tasks for the project, much less assign reasonable estimates for resources and time. If your inputs are uncertain, how can the resulting plan be anything but uncertain?

In that situation, it is better to use agile methods to learn the new technologies. Complete some small projects, perhaps for internal use, that use the tools for mobile development. Gain experience with them. Learn the hazards and understand the risks.

When you have experience, use waterfall to plan your projects. With experience behind you, your estimates will be better.

You don't have to use waterfall or agile exclusively. Some projects (perhaps many) require some exploration and research. That surveying is best done with agile methods. Once the knowledge is learned, once the team is familiar with the technology and the code, a waterfall project makes good business sense. (You have to deliver on time, don't you?)

As a manager, you have two tools to plan and manage projects. Use them effectively.

Agile is not compatible with silos

Agile development methods are very different from the traditional Waterfall methods. So different that they can affect the culture of the organization.

Agile make a different promise than Waterfall. Waterfall promises a specific deliverable on a specific date; Agile promises that you can ship whenever you want.

Agile discourages specialization. An iteration is short yet requires analysis, development, and testing. Such a short cycle does not allow for different individuals to perform different tasks.

Yet the biggest difference between Agile and Waterfall is the partitioning of tasks and the encapsulation of information. Waterfall strives for clean, discrete changes from one phase to another, with information flowing between phases in well-defined documents. The flow between the requirements phase and the development phase is the requirements document (or documents). The test results are presented in a specific document. And so on.

Information in each phase is encapsulated in that phase, and only a small set of information is allowed to transfer (one might say 'leak') to another phase.

The partitioning of tasks and the encapsulation of information leads to silos within the organization. Once separate teams are established for requirements, development, testing, and deployment, tensions arise between teams. The testing team identifies defects that reflect on the development team. The development team blames the requirements team for incomplete or ambiguous specifications.

Agile -- at least Agile for small teams -- has none of that. The fast cycles of feature selection, design, development, and test provide immediate feedback. An ambiguous requirement is spotted early, and it is obvious to everyone. Defects are identified and fixed before implementing the next feature set.

More importantly, an Agile project has one team, and the measurement of success for that team is the delivery of software. That focus on success and the inability to shift blame to another team means that it is harder to establish silos.

Which is not to say that Agile will eliminate all silos. An organization with many Agile projects can still have silos. A large company using an "Agile for large companies" process may develop silos.

But for the most part, I believe Agile processes are incompatible with silos. The involvement of necessary stakeholders; the coordinated work of design, development, and testing; and the fast cycle times all push against silo-ization.

Waterfall caused specialization; agile causes generalization

There are a number of differences between waterfall processes and agile processes. Waterfall defines one long, segmented process; agile uses a series of short iterations. Waterfall specifies a product on a specific date; agile guarantees a shippable product throughout the development process.

Another difference between waterfall and agile is the specialization of participants. Waterfall projects are divided into phases: analysis, development, testing, etc. and these phases tend to be long themselves. Agile projects have the same activities (analysis, development, testing) but on a much shorter timeframe. A waterfall project may extend for six months, or a year, or several years, and the phases of those projects may extend for months -- or possibly years.

The segmentation of a waterfall project leads to specialization of the participants. It is common to find a waterfall project staffed by analysts, developers, and testers, each a distinct team with distinct management teams. The different teams use tools specific to their tasks: databases for requirements, compilers and integrated development environments, test automation software and test case management systems.

This specialization is possible due to the long phases of waterfall projects. It is reasonable to have team "A" work on the requirements for a project and then (when the requirements are deemed complete) assign team "B" to the development phase. While team "B" develops the project, team "A" can compose the requirements for another project. Specialization allows for efficient scheduling of personnel.

Agile processes, in contrast, have short iterations of analysis, design, coding, and testing. Instead of months (or years), an iteration may be one or two weeks. With such a short period of time, it makes little sense to have multiple teams work on different aspects. The transfer of knowledge from one team to another, a small task in a multi-year project, is a large effort on a two-week iteration. Such inefficiencies are not practical for short projects, and the better approach is for a single team to perform all of the tasks. (Also, the two-week iteration is not divided into a neat linear sequence of analysis, design, development, and test. All activities occur throughout the iteration, with multiple instances of each task.)

A successful agile process needs people who can perform all of the tasks. It needs not specialists but generalists.

Years of waterfall projects have trained people and companies into thinking that specialists are more efficient than generalists. (There may be a bit of Taylorism here, too.) Such thinking is so pervasive that one finds specialization in the company's job descriptions. One can find specific job descriptions for business analysts, developers, and testers (or "QA Analysts").

The shift to agile projects will lead to a re-thinking of specialization. Generalists will be desired; specialists will find it difficult to fit in. Eventually, generalists will become the norm and specialists will become, well, special. Even the job descriptions will change, with the dominant roles being "development team members" with skills in all areas and a few specialist roles for special tasks.

Healthcare.gov was a "moonshot", but the Moon mission was not

Various folks have referred to the recent project to build and launch the healthcare.gov web site as a "moonshot". They are using the term to describe a project that:

• is ambitious in scope
• has a large number of participants
• occurs in a short and fixed time frame
• consists of a single attempt that will either succeed or fail

We in IT seem to thrive on "moonshot" type projects.

But I will observe that the NASA Moon project (the Mercury, Gemini, and Apollo missions) was not a "moonshot". NASA ran the project more like an agile project than the typical waterfall project.

Let's compare.

The NASA Moon project was ambitious. One could even call it audacious.

The NASA Moon project involved a (relatively) large number of participants, including rocket scientists, metallurgists, electrical engineers, chemists, psychologists, biologists, and radio specialists. (And many more.)

The NASA Moon project had a fixed schedule of "by the end of the decade" assigned by President Kennedy in 1961.

The NASA Moon project consisted of a number of phases, each with specific goals and each with subprojects. The Mercury flights established the technology and skills to orbit the Earth. The Gemini missions built on Mercury to dock two vehicles in space. The Apollo missions used that experience to reach the Moon.

It's this last aspect that is very different from the healthcare.gov web site project (and also very different from many IT projects). The NASA Moon program was a series of projects, each feeding into the next. NASA started with a high-level goal and worked its way to that goal. They did not start with a "master project plan" that defined every task and intermediate deliverable. They learned as they went and made plans -- sensible plans, based on their newly-won experience -- for later flights.

The healthcare.gov web site is an ambitious project. It's launch has been difficult, and shows many defects. Could it have been built in an agile manner? Would an agile approach given us a better result?

The web site must perform several major tasks: authenticate users, verify income against government databases, and display valid plans offered by insurance companies. An agile approach would have built the web site in phases. Perhaps the first phase could be allowing people to register and create their profile, the second verifying income, and the third matching users with insurance plans. But such a "phased" release might have been received poorly ("what good is a web site that lets you register but do nothing else?") and perhaps not completed in time.

I don't know that agile methods would have made for better results at healthcare.gov.

But I do know that the Moon project was not a "moonshot".

Agile project cannot overrun the budget

Some recent trade rags published stories about Agile-run projects and how they overran their budgets. This seems wrong, based on what I know about Agile methods.

First, a little history. Before "agile" projects, the dominant project method was "waterfall". Waterfall is a method for running a project. All projects need a number of tasks: analysis, design, development, and testing. The waterfall process puts these in strict sequence, with each occurring before the next phase. Thus, all analysis is performed, then all design, and then all development.

The beauty of waterfall is the schedule. The waterfall method promises a specific deliverable with a specific level of quality on a specific date.

The horror or waterfall is that it rarely delivers on schedule. It is quite common for waterfall projects to overrun their schedule. (Their cost estimates, too.)

Agile is not waterfall. The agile method takes a different approach and makes a different promise. It does not promise a specific deliverable on a specific date. Instead, it makes a promise of deliverability.

An agile project consists of a number of iterations. (Some people call these iterations "sprints". I'm not too attached to either name.) Each iteration adds a small set of features. Each iteration also uses automated tests to ensure that previous features still work. The project starts with a small set of initial features and adds features on each iteration. But the important idea is that the end of each iteration is a working product. It may not have everything you want, but everything that it has does work.

The promise of agile is that you can always deliver your product.

With the agile process, one cannot overrun a budget. (You can always stop when you want.) You can, instead, underdeliver. You may get to the end of your allotted time. You may use all of your allotted funds. But you always have something to deliver. If you run over your budget or schedule, it's because you chose to spend more time or money.

Unstoppable cannon balls and immovable posts

There is an old riddle: what happens when an unstoppable cannon ball strikes an immovable post?

The unstoppable cannon ball is a hypothetical construct. It is a cannon ball that, once in motion, stays in motion. (It's ability to defy friction comes from its hypothetical nature.)

The immovable post is also a hypothetical construct. It is a post that, once placed, cannot be moved.

While the riddle is entertaining, a real-world version of it is emerging with software development methodologies. Instead of an unstoppable cannon ball and an immovable post we have the waterfall methodology and the agile methodology. Perhaps the agile method is the equivalent of the cannon ball, as both emphasize motion. It doesn't really matter, though.

What matters is that the two methodologies are incompatible. One can develop software using one methodology, or the other, but one cannot use both. For small shops that have only one project (say a start-up), this is not an issue. Larger shops (any shop with multiple projects) must choose a methodology; you cannot use waterfall for some projects and agile methods for another. (Well, you can, provided that the two sets of projects have no interaction. But that is unlikely.)

The waterfall and the agile methods make two different promises. Waterfall promises a specific deliverable, with a specific set of features, and a specific quality, on a specific date. Agile promises that the product is always shippable (that is, very high quality), and promises to implement the most important features first, but makes no promises about a complete set of features on a specific date.

It is easy to dismiss these differences if we think of these approaches as methods of developing software. It is harder to dismiss them when we realize the truth: waterfall and agile are not methods for building software but philosophies for managing projects.

Managing projects is serious business, and is the core of most IT shops. (Projects can by managed by non-IT shops, too, but I will stick with the IT world.)

One cannot run a business (a successful business) with multiple coordinated projects that use project practices which differ to the extent that waterfall and agile differ. Waterfall, with its "big design up front" and locked-in delivery dates cannot mesh with agile, with its "choose as you go" method for selecting features. The rise of agile methods -- agile project management -- means that companies must choose one or the other. Which one is important, too, and depends on how the managers of the company want to run their company.

This schism has far-reaching effects. It can be felt in the acquisition of other companies, when one merges the management of project teams. It affects the hiring of new employees: are they familiar with the practices of the hiring company? It can even affect joint projects between companies.

I don't claim that one is better than the other. My preference is for agile project management, based on my experience and projects. It may be the right methodology for you. Or waterfall may be the better methodology for you. Managers must evaluate both and consider the benefits and costs.

My advice is to learn about both and make an informed decision.

The Agile and Waterfall Impedance Mismatch

During a lunchtime conversation with a colleague, we chatted about project management and gained some insights into the Waterfall and Agile project management techniques.

Our first insight was that an organization cannot split its project management methods. It cannot run some projects with a Waterfall process and other projects with Agile processes. (At least not if the projects must coordinate deliverables.) A company that uses Waterfall processes may be tempted to run a pilot project with Agile processes -- but if that pilot project "plugs in" to other projects, the result will be failure.

The problem is that Waterfall and Agile make two different promises. Waterfall promises a specific set of functionality, with a specific level of quality, delivered on a specific date. Agile makes no such promise;  it promises to add functionality and have a product that is always ready to ship (that is a high level of quality), albeit with an unknown set of functionality. The Agile process adds small bits of functionality and waits to get them correct before adding others -- thus ensuring that everything that has been added is working as expected, but not promising to delivery everything desired by a specific date. (I am simplifying things here. Agile enthusiasts will point out that there is quite a bit more to Agile processes.)

Waterfall processes make promises that are very specific in terms of feature set, quality, and delivery time -- but are not that good at keeping them. Hence, we have a large number of projects that are late or have low quality. Agile makes promises that are specific in terms of quality, and are good at keeping them. But the promises of the Agile processes are limited to quality; they do not propose the specifics that are promised by Waterfall.

With two different promises, it is no surprise that waterfall and agile processes don't co-exist. (There are other reasons that the two methods fail to cooperate, including the "design up front" of Waterfall and the "design as you go" of Agile.)

Our second insight was that transitioning an IT shop from Waterfall to Agile methods should not be accomplished by pilot projects.

Pilot projects are suitable to introduce people to the methods -- but those pilot projects must exist in isolation from the "regular" projects. Such projects were easy to establish in the earlier age of separate systems -- "islands of automation" -- that gave each system a measure of independence. Today, it is rare to see an IT system that exists in isolation.

Rather that use pilot projects, we like the idea of introducing ideas from Agile into the standard process used within the company. Our first choice is automated testing, the unit-level automated testing that can be performed by developers. Giving each developer the ability to run automated tests introduces them to a core practice of Agile, without creating an impedance mismatch.

After automated testing, we like the idea of allowing refactoring. Often omitted from plans for Waterfall projects, refactoring is another key practice of Agile development. Once unit tests are in place, developers can refactor with confidence.

Our third insight relates to project methods and project size. We think (and this is speculation) that Agile is better suited to small projects (and small systems) and Waterfall may be better suited to large systems. Thus, if you have large, complex systems, you may want to stay with Waterfall; if you have small systems (or even small applications) then you may want to use Agile.

We think that this relationship is correlation, not causal. That is, you can pick one side of the equation and drive the other. If you have large systems, you will end up with Waterfall. But the equation works in both directions, and you don't have to start with the size of your systems. If you use Agile methods, you will end up with smaller, collaborating systems.

Now, we realize that a lot of companies have large systems. We don't believe that by switching overnight to Agile methods will lead to smaller systems overnight. A base of large systems contains a certain amount of inertia, and requires a certain effect to redesign into smaller systems.

What we do believe is that you have a choice, that you can choose to use Agile (or Waterfall) methods (carefully, avoiding impedance mismatches), and that you can change the size of your systems. You can manage the size and complexity of your systems by selecting your processes.

These are choices that managers should accept and make carefully. They should not assume that "the system is the system" and "things are big and complicated and will always be so".