Refactoring is almost accepted as necessary

The Agile Development process brought several new ideas to the practice of software development. The most interesting, I think, is the notion of re-factoring as an expected activity.

Refactoring is the re-organization of code, making it more readable and eliminating redundancies. It is an activity that serves the development team; it does not directly contribute to the finished product.

Earlier methods of software development did not list refactoring as an activity. They made the assumption that once written, the software was of sufficient quality to deliver. (Except for defects which would be detected and corrected in a "test" or "acceptance" phase.)

Agile Development, in accepting refactoring, allows for (and encourages) improvements to the code without changing the behavior of the code (that is, refactoring). It is a humbler approach, one that assumes that members of the development team will learn about the code as the write the code and identify improvements.

This is a powerful concept, and, I believe, a necessary one. Too many projects suffer from poor code quality -- the "technical backlog" or "technical debt" that many developers will mention. The poor code organization slows development, as programmers must cope with fragile and opaque code. Refactoring improves code resilience and improves visibility of important concepts. Refactored code is easier to understand and easier to change, which reduces the development time for future projects.

I suspect that all future development methods will include refactoring as a task. Agile Development, as good as it is, is not the perfect method for all projects. It is suited to projects that are exploratory in nature, projects that do not have a specific delivery date for a specific set of features.

Our next big software development method may be a derivative of Waterfall.

Agile Development (and its predecessor, Extreme Programming) was, in many ways, a rebellion against the bureaucracy and inflexibility of Waterfall. Small teams, especially in start-up companies, adopted it and were rewarded. Now, the larger, established, bureaucratic organizations envy that success. They think that adopting Agile methods will help, but I have yet to see a large organization successfully merge Agile practices into their existing processes. (And large, established, bureaucratic organizations will not convert from Waterfall to pure Agile -- at least not for any length of time.)

Instead of replacing Waterfall with Agile, large organizations will replace Waterfall with an improved version of Waterfall, one that keeps the promise of a specific deliverable on a specific date yet adds other features (one of which will be refactoring). I'm not sure who will develop it; the new process (let's give it a name and call it "Alert Waterfall") may come from IBM or Microsoft or Amazon.com, or some other technical leader.

Yet it will include the notion of refactoring, and with it the implicit declaration that code quality is important -- that it has value. And that will be a victory for developers and managers everywhere.