Higher-level constructs can be your friends

Lee Brodie, inventor of the Forth language, once said: "I wouldn't write programs in Forth. I would build a new language in Forth, one suitable for the problem at hand. Then I would write the program in that language."

Or something like that.

The idea is a good one. Programming languages are fairly low level, dealing with small-grain concepts like 'int' and 'char'. Building a higher level of abstraction helps you focus on the task at hand, and worry less about details.

We have implemented this tactically with several programming constructs.

First were libraries: blocks of commonly used functions. All modern languages have "the standard libraries", from C to C++ to Java to C# to Python.

Object-oriented programming languages were another step, tactically. They promised the ability to "represent real-world concepts" in programs.

Today we use "domain specific languages". The idea is the same -- tactically.

I keep qualifying my statements with "tactically" because for all of our efforts, we (the programming industry, as a whole) tend to not use them. Not at a strategic level.

We use the common libraries. Any C programmer has used the strxxx() functions, along with the printf() and scanf() family of functions. C++, Java, and C# programmers use the standard-issue libraries and APIs for those languages. We're good at using what is put in front of us.

But very few projects use any of these techniques (libraries, classes, and DSLs) to create higher levels of objects. Most projects use the basic language and the constructs from the supplied framework (MFC, WPF, Struts, etc.) but build very little above these levels.

Instead of following Leo Brodie's advice, projects take the language, libraries, and frameworks and build with those constructs -- and only those constructs. The code of the application -- specifically the business logic -- is built in language-level constructs. There is (often) no effort in building libraries or classes that raise the code to a level closer to business logic.

This low-level design leads to long-term problems. The biggest problem is complexity, in the form of code that manipulates ideas at multiple levels. Code that calculates mortgage interest, for example, includes logic for manipulating arrays of numbers for payment amounts and payment dates. The result is that code is hard to understand: When reading the code, one must mentally jump up and down from one level of abstraction to another.

A second problem is the use of supplied objects for something similar. In Windows MFC programs, many systems used CString objects to hold directory and file names. This is convenient for the initial programmer, but painful for the programmers who follow. A CString object was not a file name, and had operations that made no sense for file names. (The later .NET framework provided much better support for file and path names.) Here, when reading the code, one must constantly remind oneself that the object in view is not used as a normal object of the given type, but as a special case with only certain operations allowed. This imposes work on the reader.

Given these costs of using low-level constructs, why do we avoid higher-level constructs?

I have a few ideas:

Building higher-level constructs is hard: It takes time and effort to build good (that is, useful and enduring) constructs. It is much easier (and faster) to build a program with the supplied objects.

Constructs require maintenance: Once "completed", constructs must be modified as the business changes. (Code built from low-level objects needs to be modified too, but managers and programmers seem more accepting of these changes.)

Obvious ramp-up time: Business-specific high-level constructs are specific to that business, and new hires must learn them. But with programs built with low-level constructs, new hires can be productive immediately, since the system is all "common, standard" code. (Systems have non-obvious ramp-up time, as new hires "learn the business", but managers seem to accept -- or ignore -- that cost.)

Can create politics: Home-grown libraries and frameworks can create political conflicts, driven by ego or overly-aggressive schedules. This is especially possible when the library becomes a separate project, used by other (client) projects. The manager of the library project must work well with the managers of the client projects.

These are not technical problems. These challenges are to the management of the projects. (To some extent, there are technical challenges in the design of the library/class/framework, but these are small compared to the managerial issues.)

I still like Leo Brodie's idea. We can do better than we currently do. We can build systems with better levels of abstraction.