Relearning the system design lessons of history

The scene: a local user group with a presentation on system design for Big Data.

The participants: technologists of varying degrees of experience.

The presenter made a number of points. Two were: separate data retrieval from processing, and design systems with low coupling between components.

It struck me that these two points had been made before. Not at previous meetings of this group, but in the IT industry.

Designing systems with low coupling between components has been around for decades, going back to the 1970s. Yourdon and Demarco both talk about low coupling (and high cohesion), as do others.

Separating data retrieval from processing has been around even longer, with IBM's "HIPO" methods (Hierarchy, Input, Process, and Output) for system design.

It seems that every generation of programmers re-discovers the concepts of prior generations.

And that though got me thinking.

A long time ago, in a galaxy (of hardware) far, far away, I was a young upstart. I was learning to program, learning to design systems, and learning to build applications. I was using microcomputers, the immediate predecessors to the IBM PC. The most common computers of the era were the Apple II, the Radio Shack TRS-80, and the Commodore PET. There were other computers; most were S-100 buss computers that ran CP/M.

We had little in the way of software. Programs were written in BASIC, or in assembly language. There were a few other options, usually expensive: FORTRAN, COBOL, or CBASIC.

Hobbyists and enthusiasts would band together in groups, to review new ideas and discuss problems. Some of us had experience in the older systems (minicomputers or mainframes) but not many. We fumbled our way to knowledge without the immediate aid of our predecessors.

In a word, we were arrogant. We felt that the programmers of mainframes and minicomputers would have little to offer us. (After all, they insisted on using old languages and old machine architectures. We were using shiny new processors and better languages.) We did little (if anything) to reach out to the programmers of the elder systems.

I think that the arrogance was not limited to us upstarts. The programmers of mainframes and minicomputer systems did little to reach out to us microcomputer users. (At least, I got that impression from the magazines devoted to mainframes and minicomputers, and their lack of coverage on microcomputers.)

But then, my perception may be incorrect. Perhaps the mainframe and minicomputer folks (and their magazines) were busy with issues of the day, busy with solutions relevant to business problems. Microcomputers were not a business problem, nor were they a solution to a problem. They were ignoring us not from malice but from pragmatism.

And perhaps that is the way our industry works. A lot of our systems are complex and a lot of our problems are hard. We tend to think that we are alone, and that we must devise a solution. And devise solutions we do.

Not only do we devise solutions, but we name them. We name lots of things, from programming languages to development methodologies. Each generation invents new terms. We often have generation-specific terms for identical or similar concepts (we "boot" PCs, but we "IPL" mainframes; PC programs "crash", mainframe programs "abend"). The different terms are barriers to communication.

In addition, we become tightly focused on set of practices and technologies. Programmers who know C++ work on C++ systems and become experts at it. Programmers in Java work on Java systems and become experts. Many programmers specialize; few generalize. Specialization begets jargon and argot, and additional barriers to communication.

The solution (and I am not the first to propose this) is a broad curriculum for software developers. The curriculum includes current technologies (mobile apps, Big Data) mainstream technologies (web apps and services), and legacy technologies (desktop). It includes general concepts (data structures, compilers) and specific knowledge (object-oriented programming). And most importantly, it includes history, to give people the knowledge that lots of current-day problems have been solved, albeit in other contexts.

As I see it, we can learn the lessons of history up front, or we can learn them later. But we cannot escape them.

I'm for learning those lessons up front.