Discipline in programming

Programming has changed over the years. We've created new languages and added features to existing languages. Old languages that many consider obsolete are still in use, and still changing. (COBOL and C++ are two examples.)

Looking at individual changes, it is difficult to see a general pattern. But stepping back and getting a broader view, we can see that the major changes have increased discipline and rigor.

The first major change was the use of high-level languages in place of assembly language. Using high-level languages provided some degree of portability across different hardware (one could, theoretically, run the same FORTRAN program on IBM, Honeywell, and Burroughs mainframes). It meant a distant relationship with the hardware and a reliance on the compiler writers.

The next change was structured programming. It changed our notions of flow control, using "while", "if/then/else", and "for" structures and discouraged the use of "goto".

Then we adopted relational databases, separate from the application program. It required using an API (later standardized as SQL) rather than accessing data directly, and it required thought and planning for the database.

Relational databases forced us to organize data stored on disk. Object-oriented programming forced us to organize data in memory. We needed object models and for very large projects, separate teams to manage the models.

Each of these changes added discipline to programming. The shift to compilers required reliable compilers and reliable vendors to support them. Structured programming applied rigor to the sequence of computation. Relational databases applied rigor to the organization of data stored outside of memory, that is, on disk. Object-oriented programming applied rigor to the organization of data stored in memory.

I should note that each of these changes was opposed. Each had naysayers, usually basing their arguments on performance. And to be fair, the initial implementation of each change did have lower performance than the old way. Yet each change has a group of advocates (I call them "the Pascal crowd" after the early devotees to that language) who pushed for the change. Eventually, the new methods were improved and accepted.

The overall trend is towards rigor and discipline. In other words, the Pascal crowd has consistently won the debates.

Which is why, when looking ahead, I think future changes will keep moving in the direction of rigor and discipline. There may be minor deviations from this path, with new languages introducing undisciplined concepts, but I suspect that they will languish. The successful languages will require more thought, more planning, and prevent more "dangerous" operations.

Functional programming is promising. It applies rigor to the state of our program. Functional programming languages use immutable objects, which once made cannot be changed. As the state of the program is the sum of the state of all variables, functional programming demands more thought given to the state of our system. That fits in with the overall trend.

So I expect that functional languages, like structured languages and object-oriented languages, will be gradually adopted and their style will be accepted as normal. And I expect more changes, all in the direction of improved rigor and discipline.

Our technology shapes our systems

In the old days, computer programs were fairly linear things. They processed data in a linear fashion, and the source code often appeared in a linear fashion.

Early business applications of computers were for accounting applications: general ledger, accounts payable, payroll, inventory... etc. These systems were often designed with a master file and one or more transaction files. The master file held information about customers, accounts, and inventory, and the transaction files held information about, well, transactions, discrete events that changed something in the master file. (For example, a bank's checking accounts would have balances in the master file, and records in the transaction file would adjust those balances. Deposits would increase a balance, checks or fees would decrease a balance.)

The files were not stored on modern devices such as USB drives or even floppy disks. In the early days, the master file was on magnetic tape, and the transactions were on punch cards.

The thing about magnetic tape is that you must run through it from beginning to end. (Much like a tour through an Ikea store.) You cannot jump around from one position to another; you must start with the first record, then process the second record, and in sequence process every record until the last.

The same holds for punch cards. Paper punch cards were placed in a hopper and read and processed one at a time.

You might wonder how you can handle processing of accounts with such restrictions in place. One pass through the master file? And only one pass through the transactions? How can we match transactions to master records if we cannot move to the proper record?

The trick was to align the input files, keeping the master file sorted and sorting the transactions before starting the update process. With a bit of thought, you can imagine a system that reads a master record and a transaction record, compares the account numbers on each (both records need a key for matching) and if they match then updates the master record and moves on to the next transaction. If they don't match then the system stores the master record (on another tape, the output tape) and runs the comparison again. The algorithm does work (although I have simplified it somewhat) and this was a common model for program design.

The rise of direct-access storage devices and complex data structures has changed programming. As processors became less expensive and more powerful, as programming languages became more expressive and allowed complex data structures such as lists and trees, as memory became available to hold complex data structures in their entirety, our model for programming became more complex. No longer were programs limited to the simple cycle of "read-master, read-transaction, compare, update, write-master, repeat".

Programming in that model (perhaps we could call it the "Transaction Pattern") was easy and low-risk because clever people figured out the algorithm and other people could copy it.

This notion of a common system model is not unique to 1960s-style programming. Microsoft Windows programs at the API level follow a specific pattern of messages sent by the Windows core "message pump". Android programs use a similar technique.

Tablet/cloud systems will probably develop one (or perhaps a handful) of common patterns, repeated (perhaps with some variations) for the majority of applications. The trick will be to identify the patterns that let us leverage the platform with minimal thought and risk. Keep your eyes open for common templates for systems. When you find one that works, when you find one that lets lots of people leverage the cleverness of a few individuals, stick with it.

I'm guessing that the system model will not be a purely linear one, as we had in the 1960s. But it may have linear aspects, with message queues serializing transactions and updates.