From the bottom up

Fixing software is hard. Experienced managers of development projects have memories (often painful memories) of projects to redesign a system. Redesign efforts are hard to justify, difficult to plan, and risk breaking code that works. All too often, a redesign project ends late, over budget, and with code that is just as hard to maintain. The result is the trifecta of project failures.

A big part of the failure is the approach to redesign. Projects to fix code often use one of the following approaches:

- Re-write the entire program (or system)
- Improve code along functional components (the "room by room" approach)
- Improve code as you modify it ("fix it as we make other changes")

Re-writing the system is expensive. Often too expensive to justify. ("You're going to spend how many dollars and how many months? And the result will be the same as what we have now?" ask the senior managers. And there is no good answer.) Plus, you have the risk of missing important functionality.

Changing the system in parts, akin to replacing tires on a car, is appealing but never seems to work. Components have too many connections to other components, and to truly fix a component one must fix the other components... and you're back to re-writing the entire system. (Except now you don't have the money or the budget for it.)

Fixing code as you make other changes (as you are implementing new functionality) doesn't work. It also has the "dependency effect" of the component approach, where you cannot fix code in one module without fixing code in a bunch of other modules.

Managers are poor at planning redesign projects, because they manage software like they manage the organization: from the top down. While top-down works for people, it does not work for code.

People organizations are, compared to software, fairly simple. Even a large company such as Wal-mart with its one million employees is small, compared to software projects of many million lines of code. Lines of code are not the same as people, but the measures are reasonable for comparison. Any line of code may "talk" with any other line of code; that is not true for most organizations.

People organizations are, compared to software, fairly smart. People are sentient beings and can act on their own. A corporate manager can issue directives to his reports and expect them to carry out orders. Software, on the other hand, cannot re-organize itself. Except for a few specialized AI systems, software must be guided.

I have worked on a number of projects. In each case, the only method that worked -- really worked -- was the "bottom up" approach.

In the bottom up approach, you start with the smallest components (functions, classes, whatever makes up your system), redesign them to make them "clean", and then move upwards. And to do this, you must know about your code.

You start with a survey the entire code. You need this survey to build a tree of the dependencies between functions and classes. Use the tree to identify the code at the bottom of the tree -- code that stands alone and does not depend on other code (except for standard libraries). That is the code to redesign first. Once you fix the bottom code, move up to the next layer of code -- code that depends only on the newly-cleaned code. Repeat this process until you reach the top.

Why the "bottom up" method for software?

Software is all about dependencies. Fixing software is more about changing dependencies than it is fixing code. Redesigning software changes the dependencies; improving code in the middle of a system will affect code above and below. Changes can affect more areas of the code, like ripples spreading across the surface of a lake. Starting at the bottom lets us change the code and worry about the ripples in only one direction.

Why not start at the top (with ripples in only one direction)? Changes at the top run the risk of leading us in the wrong direction. We think that we know how the software should be organized. But this impression is wrong. If we have learned anything from the past sixty years of software development, it is that we do not know, up front, how to organize software. If we start at the top, we will most likely pick the wrong design.

On the other hand, starting at the bottom lets us redesign the smallest components of the system, and leverage our knowledge of their current uses within the system. That extra knowledge lets us build better components. Once the smallest components are "clean", we can move to the next level of components and redesign them using knowledge of their use and clean subcomponents.

If anything, starting from the bottom reduces risk. By working on small components, a team can improve the code and make definite gains on the redesign of the system. The smaller objectives of bottom-up redesign are easier to accomplish and easier to measure. Once made, the changes are a permanent part of the system, easing future development efforts.

Software is not people; it does not act like people and it cannot be managed like people. Software is its own thing, and must be managed properly. Redesign projects from the top down are large, expensive, and risky. Projects from the bottom up have a better chance of success.