Two-cycle development

I have been working with software for a while now (more years than I care to mention) and I have found that I work in two different modes. The first mode is 'creating software' and the second is 'refining software'.

The 'creating' mode is when I add new features to a program. It could be at the very beginning of a development effort, when there are no features in a program, or it could be later, when the software is 'mature' (whatever that means).

The 'refining' mode is different. It is not adding features, but changing the software to perform the same feature set but with better internal design. This might mean replacing a sort routine with a better sort routine, refactoring a class for better organization, or changing a program for better division of labor. The important part in this phase is that no features change -- the program works just as before. The internals are different, the externals are the same.

These two modes of thought are, in my mind, part of the basic human psyche. At some times, our minds want to create. At other times, our minds want to re-arrange that which we have created. I consider both modes creative, although different types of creativity.

I think that two-cycle development is necessary. It's necessary because we don't get the design of the program right the first time. Oh, we get something that works, but that doesn't mean the design is right. Despite all of the planning, all of the analysis, and all of the reviews, we build software poorly. Refactoring allows us to examine our construct, think about the design, and figure out better ways to do the job.

Two-cycle development leverages our 'Monday morning quarterbacking' talents (even for our own work), and is possible with automated tests. The ability to re-arrange, to experiment, to try new things, is much easier with a complete set of tests. You need the tests to verify your experiment. Without automated tests, you must rely on manual tests, and manual tests are hard to perform. Since they are hard to perform, programmers tend to skip them. And when programmers skip the tests, they miss errors. After a few experiments in re-organizing code without tests (and a few introduced defects) programmers learn to avoid refactoring operations.

Without automated tests, development tends to avoid refactoring.

Without the refactoring, we use our initial, poor designs. While they work, the designs fail over time. And they often fail in slow ways. Failures occur not in spectacular collapses, but in slow decay. Any system built without refactoring gradually accumulates poor designs, and these poor designs increase the future work on the program. Over time, changes to the program take more time and effort, and the number of defects per change increases. The program gets tangled and hard to understand. Eventually, development grinds to a halt as each change introduces more defects than improvement.

To prevent this 'slow death by poor design', we must allow (and plan for) refactoring. And to refactor the program, we must do to things: allow developers time for refactoring and provide the tests to verify that the changes work. Without these, the program accumulates 'cruft' and becomes harder to maintain.