Closing the "Iron Triangle"

Microsoft's new releases of Outlook and Media Player highlight a coming crisis for software development. There are two problems, caused by the closing the "Iron Triangle" of software development.

The Iron Triangle is a phrase to succinctly state that software development requires trade-offs among  three aspects: cost, quality, and performance. It is easy to build software that is cheap -- as long as it doesn't have to work right and doesn't have to be fast. It is easy to build software that is correct -- as long as it doesn't have to be fast and the budget is high enough. It is easy to build software cheaply -- as long as you don't care about quality or performance. It is easy to meet one of the three aspects.

It's somewhat harder (but still easy) to work within two aspects (say, cost and quality) while not caring about the third (performance, in this case).

Working with constraints for all three aspects (cost, quality, and performance) is difficult. Extremely difficult. More on this later.

Back to Microsoft and its new versions of Outlook and Media Player.

The first problem is that both the new Outlook and the new Media Player are slower and use more memory than the previous versions. (The new Media Player also wants payment for some previously free codecs, but that is a separate issue.)

I told a fib there. The problem is not that new Outlook and new Media Player have lower performance. The problem is that people have noticed. People are now aware of changes to performance, probably because they have  already planned to delay hardware upgrades. (Possibly due to higher prices or lack of availability. Possibly because they are reluctant to upgrade after recent upgrades for Windows 11's higher requirements for hardware.)

The second, and larger, problem is that Microsoft has long relied on Moore's Law. (Moore's Law states that the size of a transistor halves every eighteen months. It can roughly be read as "computing power for CPU and memory doubles every two years".) Microsoft is not alone in this.

Relying on Moore's Law means that development shops can design and build software that is large and slow. Instead of designing lightweight, efficient software, shops can build heavy software with poor performance, and then wait two years. The next generation of computers will be faster and the performance will be acceptable (on the new, faster hardware).

It's a nice arrangement, and it works... until it doesn't.

The rate of reduction of transistor sizes has slowed. Over time we learn to build smaller transistors, but not at the rate of "half the size every eighteen months". Effectively, the time to double performance has increased, which means that users cannot simply wait two years for better performance.

If the size of transistors is not enough, the demand for hardware to build AI data centers has affected the cost and availability of PCs. Costs of "regular" PCs are up. Availability of memory and disk drives (either SSD or HDD) are down, and their costs are up. It's not just that new PCs are not as "faster" as they used to be, it's that PCs are more expensive and only slightly faster than they used to be. Consequently, PCs aren't replaced or upgraded as often as they used to be.

All of this means that the hardware used by customers is stagnant. It's not improving in terms of performance.

If the providers of software, either PC apps or web apps (because web apps run on PC browsers), want to add features, they had better do it without slowing their software. Microsoft is learning this lesson now, with Outlook.

And that brings us to the bigger of the two problems: Development shops don't know how to build lightweight, efficient software.

Software development for the past half-century has been able to rely on Moore's Law and increasing hardware performance. They have spent that time learning how to build software that is reliable. They have spent that time learning how to build software that is readable and maintainable. And they have spent some of that time learning how to build software that is secure. We have invented programming languages to let us build better software: Java for reliability, C# for readability, Rust for security.

With the focus on reliability and readability and security, there has been precious little time for lightweight design. Little time for performance. Consequently, just about all of the knowledge we had for performance in software has retired with the grey-haired programmers from the mainframe age.

It's not merely individuals. Organizations no longer have the knowledge (or even the desire) to build software that performs well. Teams that create the specifications for software don't think about performance. Teams for code design and code review don't think about it, Teams for testing don't think about it.

Side note: There are some teams, in specialized contexts, that do care and think about performance. Embedded systems, telephone switch systems, financial transaction systems, and databases all need high performance. Those teams will be aware of performance and efficiency. But for the rest of us, there hasn't been a need for performance because we have relied on hardware improvements.

Second side note: Ignoring performance for the past half century was a rational choice. Building software that performs well requires time (which is money) and resources (which pull away from new features). New features provided a competitive advantage; faster software did not. Organizations simply followed the money.

Now we have to think about performance.

The first challenge is getting organizations to recognize that performance is important. Some will recognize this on their own; many will see it when customers start complaining. Once they realize that performance is important, different organizations may react differently.

Some organizations may ignore it, hoping that the plateauing of performance is temporary and that advances in hardware will resume. Naive organizations may recognize the need to improve performance and simply add performance requirements to the input of their SDLC (Software Development Life Cycle) and think that the job is done. Organizations that are aware of the intricacies of performance will understand that they face a difficult challenge.

Adding performance closes the Iron Triangle. Without performance requirements, software could be developed relatively easily. The development team could trade off performance for faster delivery, or higher quality. Relying on Moore's Law left one side of the triangle open, essentially a free pool that could absorb the costs of development decisions.

With that side closed, developers must now find solutions which live within the triangle. They cannot use performance as a free token to trade away. They cannot implement new features (with new internal data structures and additional database calls) without considering and working around the performance cost of those structures and calls.

Writing software that performs well is an art. (So is writing software that is readable, and writing software that is correct.) But writing software that is readable, and correct, and performs well is extremely difficult. Many times it is not possible. Compromises must be made to live within the triangle of constraints. If adding a feature requires additional processing, that processing takes time. If adding a feature requires database calls, those database calls take time. If reducing the time for database calls requires more memory (to hold results in a cache) then one gets better performance at the expense of memory. (And some risk, as cache invalidation is a difficult problem.) There are many ways to trade one aspect (performance, cost, quality) for another, but they are trade-offs and one aspect gains while others lose.

How good are organizations at making those trade-offs? How good are development teams at presenting the choices for compromise? How willing are managers to accept the trade-offs provided by the development team and offering guidance for solutions? (Or do managers simply reject the request and mandate a delivered system on the due date?)

With the shortage of hardware and the slowness of hardware improvements, organizations will have to learn to build lightweight software. Doing so requires that they close the Iron Triangle and add performance requirements. Adding those requirements will necessitate communication about software design (more than there is now) and decisions that trade features, quality, and performance. Those conversations require an understanding of the issues and a willingness to compromise.

It is the organizations that prepare for and hold these conversations that will provide the better products.

Apple hardware is outpacing Apple software

Something interesting about the new iPhone 16: the software isn't ready. Specifically, the AI ("Apple Intelligence") enhancements promised by Apple are still only that: promises.

Apple can develop hardware faster than it can develop software. That's a problem.

Apple has had this problem for a while. The M1 Mac computers first showed this problem. Apple delivered the computers, with their integrated system-on-chip design and more efficient processing, but delivered no software to take advantage of that processing.

It may be that Apple cares little for software. They sell computers -- hardware -- and not software. And it appears that Apple has "outsourced" the development of applications: it relies on third parties to build applications for Macs and iPhones. Oh, Apple delivers some core applications, such as utilities to configure the device, the App Store to install apps, and low-powered applications such as Pages and Numbers. But there is little beyond that.

Apple's software development focusses on the operating system and features for the device: MacOS and Pages for the Mac, iPadOS and Stage Manager for the iPad, iOS and Facetime and Maps for the iPhone. Apple builds no database systems, has lukewarm support and enhancements for the Xcode IDE, and few apps for the iPhone.

I suspect that Apple's ability to develop software has atrophied. Apple has concentrated its efforts on hardware (and done rather well) but has lost its way with software.

That explains the delay for Apple Intelligence on the iPhone. Apple spent a lot of time and effort on the project, and (I suspect) most of that was for the hardware. Updates to iOS for the new iPhone were (probably) fairly easy and routine. But the new stuff, the thing that needed a lot of work, was Apple Intelligence.

And it's late.

Thinking about the history of Apple's software, I cannot remember a similar big feature added by Apple. There is Facetime, which seems impressive but I think the iPhone app is rather simple and a lot of the work is in the back end and scalability of that back end. Stage Manager was (is) also rather simple. Even features of the Apple Watch such as fall detection and SOS calls are not that complex. Operating systems were not that difficult: The original iOS was new, but iPadOS is a fork of that and WatchOS is a fork of it too (I think).

Apple Intelligence is a large effort, a greenfield effort (no existing code), and one that is very different from past efforts. Perhaps it is not surprising that Apple is having difficulties.

I expect that Apple Intelligence will be delivered later than expected, and will have more bugs and problems than most Apple software.

I also expect to see more defects and exploits in Apple's operating systems. Operating systems are not particularly complex (they are as complex as one makes them) but development and maintenance requires discipline. One gets that discipline through constant development and constant monitoring of that development. It requires an appreciation of the importance of the software, and I'm not sure that Apple has that mindset.

If I'm right, we will see more and more problems with Apple software. (Slowly at first, and then all at once.) Recovery will require a change in Apple's management philosophy and probably the senior management team.

To build and to maintain

I had the opportunity to visit another country recently (which one doesn't matter) and I enjoyed the warmer climate and the food. I also had the opportunity to observe another country's practices for building and maintaining houses, office buildings, roads, bridges, and other things.

The United States is pretty good at building things (roads, bridges, buildings, and such) and also good at maintaining them. The quality of construction and the practices for maintenance vary, of course, and overall governments and large corporations are better at them than small companies or individuals.

In the country I visited, the level of maintenance was lower. The culture of the country is such that people in the country are good at building things, but less concerned with maintaining them. This was apparent in things like signs in public parks: once installed they were left exposed to the elements where they faded and broke in the sun and wind.

My point is not to criticize the country or its culture, but to observe that maintaining something is quite different from building it.

That difference also applies to software. The practices of maintaining software are different from the practices of constructing software.

Software does not wear or erode like physical objects. Buildings expand and contract, develop leaks, and suffer damage. Software, stored in bits, does not expand and contract. It does not develop leaks (memory leaks aside), It is impervious to wind, rain, and fire. So why do I say that software needs maintenance?

I can make two arguments for maintenance of software. The first argument is a cyber-world analog of damage: The technology platform changes, and sometimes the software must change to adapt. A Windows application, for example, may have been designed for one version of Windows. Windows, though, is not a permanent platform; Microsoft releases new versions with new capabilities and other changes. While Microsoft makes a considerable effort to maintain compatibility, there are times when changes are necessary. Thus, maintenance is required.

The second argument is less direct, but perhaps more persuasive. The purpose of maintenance (for software) is to ensure that the software continues to run, possibly with other enhancements or changes. Yet software, when initially built, can be assembled via shortcuts and poor implementations -- what we commonly call "technical debt". Often, those choices were made to allow for rapid delivery.

Once the software is "complete" -- or at least functional, maintenance can be the act of reducing technical debt, with the goal of allowing future changes to be made quickly and reliably. This is not the traditional meaning of maintenance for software, yet it seems to correspond well with the maintenance of "real world" objects such as automobiles and houses. Maintenance is work performed to keep the object running.

If we accept this definition of maintenance for software, then we have a closer alignment of software with real-world objects. It also provides a purpose for maintenance; to ensure the long-term viability of the software.

Let's go back to the notions of building and maintaining. They are very different, as anyone who has maintained software (or a house, or an automobile).

Building a thing (software or otherwise) requires a certain set of skills and experience.

Maintaining that thing requires a different set of skills and experience. Which probably means that the work for maintenance needs a different set of management techniques, and a different set of measurements.

And building a thing in such a way that it can be maintained requires yet another set of skills and experience. And that implies yet another set of management techniques and measurements.

All of this may be intuitively obvious (like solutions to certain mathematics problems were intuitively obvious to my professors). Or perhaps not obvious (like solutions to certain math problems were to me). In either case, I think it is worth considering.

What Agile and Waterfall have in common

Agile and Waterfall are often described in contrasts: Waterfall is large and bureaucratic, Agile is light and nimble. Waterfall is old, Agile is new. Waterfall is... you get the idea.

But Waterfall and Agile have things in common.

First and most obvious, Waterfall and Agile are both used to manage development projects. They both deliver software.

I'm sure that they are both used to deliver things other than software. They are tools for managing projects, not limited to software projects.

But those are the obvious common elements.

An overlooked commonality is the task of defining small project steps. For Waterfall, this is the design phase, in which requirements are translated into system design. The complete set of requirements can paint a broad picture of the system, providing a general shape and contours. (Individual requirements can be quite specific, with details on input data, calculations, and output data.)

Breaking down the large idea of the system into smaller, code-able pieces is a talent required for Waterfall. It is how you move from requirements to coding.

Agile also needs that talent. In contrast to Waterfall, Agile does not envision the completed system and does not break that large picture into smaller segments. Instead, Agile asks the team to start with a small piece of the system (most often a core function) and build that single piece.

This focus on a single task is, essentially, the same as the design phase in Waterfall. It converts a requirement (or user story, or use case, or whatever small unit is convenient) into design for code.

The difference between Waterfall and Agile is obvious: Waterfall converts all requirements in one large batch before any coding is started, and Agile performs the conversions serially, seeing one requirement all the way to coding and testing (or more properly, testing and then coding!) before starting the next.

So whether you use Waterfall or Agile, you need the ability to "zoom in" from requirements to design, and then on to tests and code. Waterfall and Agile are different, but the differences are more in the sequence of performing tasks and not the tasks themselves.

Parallel processing on the horizon

Parallel processing has been with us for years. Or at least attempts at parallel processing.

Parallel processing has failed due the numerous challenges it faces. It requires special (usually expensive) hardware. Parallel processing on convention CPUs is simply processing items serially, because conventional CPUs can process only serially. (Multi-core processors address this problem to a small degree.) Parallel processing requires support in compilers and run-time libraries, and often new data structures. Most importantly, parallel processing requires tasks that are partitionable. The classic example of "nine women producing a baby in one month" highlights a task that is not partitionable, not divisible, into smaller tasks.

Cloud computing offers a new twist on parallel processing.

First, it offers multiple processors. Not just multiple cores, but true multiple processors -- as many as you would like.

Second, it offers these processors cheaply.

Cloud computing is a platform that can handle parallel processing -- in some areas. It has its problems.

First, creating new cloud processing systems is expensive in terms of time. A virtual machine must be instantiated, started, and given software to handle the task. Then, data must be shipped to the server. After processing, the result must be sent back, or forward to another processor. The time for all of these tasks is significant.

Second, we still have the problems of partitioning tasks and representing the data and operations in a program.

There is one area of development that I believe is ready to leverage parallel processing. That area is testing.

The typical testing effort for a project can have multiple levels: unit tests, component tests, system tests, end-to-end tests, you name it. But each level of testing follows the same general pattern:

• Get a collection of tests, complete with input data and expected results
• For each test
• 1) Set up a test environment (program and data)
• 2) Run the test
• 3) Compare output to expected output
• 4) Record the results
• Summarize the results and report

In this process, the sequence of steps I've labelled 1 through 4 is repeated for each test. Traditional testing puts all of these tests on one computer, performing each test in sequence. Parallel testing can put each test on its own cloud-based processor, effectively running all tests at once.

Testing has a series of well-defined and partitionable tasks. Modern testing methods use automated tests, so a test can run locally or remotely (as long as it has access to everything it needs). Testing can be a drain on resources and time, requiring lots of requests to servers and lots of time to complete all tests.

Testing in the cloud, and in parallel, addresses these issues. It reduces the time for tests and improves the feedback to developers. Cloud processing is cheap -- at least cheaper than paying developers to wait for tests to run.

I think one the next "process improvements" for software development will be the use of cloud processing to run tests. Look for new services and changes to testing frameworks to support this new mode of testing.

The provenance of variables

Just about every programming language has the concept of a 'variable', a container of a value. Variables are named 'variable' because their contents can vary -- as opposed to constants. The statement

a = 10

Assigns the variable 'a' a value of '10', denoted in the program as a constant.

(There are some languages which allow for the values of constants to be changed, so one can assign a new value to a constant. It leads to unusual results and is often considered a defect. But I digress.)

The nice thing about variables is that they can vary. The problem with variables is that they vary.

More specifically, when examining a program (say with a debugger), one can see the contents of a variable but one does not know how that value was calculated. Was it assigned? Was the variable incremented? Did the value come from a constant, or was it calculated? When was it assigned?

Here is an idea: Retain the source of values. Modify the notion of a variable. Instead of being a simple container for a value, hold the value and additional information.

For example, a small program:

file f = open("filename")
a = f.read()
b = f.read()
c = (a - b) / 100

Let's assume that the file contains the text "20 4", which is the number 20 followed by a space an then the number 4, all in text format.

In today's programming languages, the variables a, b, and c contains values, and nothing else. The variable 'a' contains 20, the variable 'b' contains '4', and the variable 'c' contains 0.16. Yet they contain no information about how those values were derived.

For a small program such as this example, we can easily look at the code and identify the source of the values. But larger programs are a different story. They are often complex, and the source of a value is not obvious.

With provenance, the variables a, b, and c still contain values, and in addition contain information about those values.

The variable 'a' contains the value 20 and the value 'filename', as that was the source of the value. It would also be possible to contain more information about the file, such as a creation date, a version number (for filesystems that support version numbers), and the position within the file. It can even contain the line number of the assignment, allowing the programmer easy access to the source.

The variable 'b' contains similar information.

The variable 'c' contains information about the variables 'a' and 'b', along with their states at the time of assignment. Consider the revised program:

file f = open("filename")
a = f.read()
b = f.read()
c = (a - b) / 100
... more code
a = 0
b = 1
... more code
d = c * 20

In this program, the variable 'd' is assigned the value 3.2 (assuming that the same file is read) and at that assignment, the variable 'c' holds information about 'a' and 'b' with their initial values of 20 and 4, not their current values of 0 and 1. Thus, a developer can examine the assignment to 'd' and understand the value of 'c'.

In addition to developers, provenance may be useful for financial auditors and examiners. Anyone who cares about the origins of a specific value will find provenance helpful.

Astute readers will be already thinking of the memory requirements for such a scheme. Retaining provenance requires memory -- a lot of memory. A simple variable holding an integer requires four bytes (on many modern systems). With provenance, a 'simple' integer would require the four bytes for the value and as many bytes as required to hold its history. Instead of four bytes, it may require 40, or 400.

Clearly, provenance is not free. It costs memory. It also costs time. Yet the benefits, I think, are clear. So, how to implement it? Some ideas:

- Provenance is needed only when debugging, not during production. Enable it as part of the normal debug information and remove it for 'release' mode.
- Provenance can be applied selectively, to a few variables and not to others.
- Provenance can be implemented selectively. Perhaps one needs only a few pieces of information, such as line number of assignment. Less information requires less memory.

Our computing capacity continues to grow. Processor capabilities, memory size, and storage size, are all increasing faster than program size. That is, our computers are getting bigger, and they are getting bigger faster than our programs are getting bigger. All of that 'extra' space should do something for us, right?

Use the tools available

I've just completed some work on a small project. My success is due, not only to my own talent and hard work, but to the tools that were available to me.

The project was in Ruby, and the tool that assisted me was Rubocop.

Rubocop analyzes Ruby code and reports on questionable (but legal) constructs and syntax. It is, in a phrase, "lint" for Ruby.

Almost all of the major languages have syntax checkers. For C and C++, there is lint. For C# there is FXCop. For Python, PyLint. Even Perl has Perl::Critic and Perl::Lint.

Rubocop helped me, indirectly. I used it as I developed the project. I ran it on the code I was writing, and it reported that certain functions were "too long", according to its default guidelines.

Some programmers would be arrogant and refuse to heed such advice. (Myself at an earlier point in my career, for example.) But with the wisdom of experience, I chose to modify the code and reduce the size of functions. It was an investment.

When I modified the long functions, I broke them into smaller ones. This had the benefit of making duplicate code obvious, as some of the smaller functions performed identical tasks. (The large versions of the functions also performed these identical tasks, but the duplications were not apparent.)

I combined duplicate functions into single functions, and reduced the overall size of the code. I also created abstract classes to hold functions common to concrete derived classes.

The simpler version of the code was, well, simpler. That meant that subsequent changes were easier to implement. In fact, one particular feature had me worried, yet the simpler code made that feature easy to add.

Rubocop helped me simplify the code, which made it easy for me to add new features -- and get them right. Rubocop was a tool, a useful tool.

On your projects, be aware of the tools that can help you.

Maintenance may be more than we think

The effort for systems is often split into "development" and "maintenance". Development is considered the premium of the two: creating a new system to address a business need; maintenance is often considered the "dirty work" of minor adjustments and corrections.

I've been thinking about the terms "development" and "maintenance", and I think our ideas need to change.

Let's consider a non-software example. Let's suppose that we build a hotel in a remote tropical location, on beachfront property. It's a simple hotel with decent rooms and a basic restaurant, but nothing special. With our current understanding of "development" and "maintenance", the construction of such a hotel is clearly "development".

Our hotel is isolated, and far away from other hotels and restaurants.

There are clearly maintenance tasks: cleaning the rooms and hallways, stocking the refrigerators (perhaps one considers that task part of "operations"), and repairing the corrosion from the salty air.

A major change, such as adding air conditioning to the guest rooms or a swimming pool in the common area, is considered "development".

But suppose that we don't add air conditioning or a swimming pool. Suppose we keep our simple hotel, but others buy property nearby and build hotels of their own. (Other people recognize the business opportunity we are enjoying.) And suppose further that they build hotels with air conditioning and swimming pools. We lose some business to the newer, fancier hotels in the area.

Now we're in a different situation. Faced with competition, we're forced to take action. Adding a swimming pool or air conditioning to our hotel is not development, at least not in the sense of defining a new service or market. Enhancing our resort to match the competition is less a matter of expansion and more a matter of stemming the loss of business. In this case, adding a swimming pool is "maintenance".

Development gets you into the market, and maintenance keeps you in the market.

Taking this view of development and maintenance for programs, maintenance becomes a task much larger than simply fixing defects and making minor changes. Maintenance is the act of staying competitive. That includes corrections of defects, and it includes major features to match the competition. It includes adjustments to match changes to the operating system, such as the GUI changes for Windows 8.

Which means that maintenance more than our traditional idea. In one sense it is the old idea of keeping something running. The old sense was focussed on the the thing; the new sense looks at the thing (the program, the hotel, etc.) and the environment. We have to keep the thing running in a changing environment.

One can argue about enhancements to a computer system (or a hotel) and the classification into the categories of "development" and "maintenance". There may be tax ramifications for that classification -- especially for "maintenance contracts". I'm ignoring the tax issues here.

Development and maintenance are perhaps not as distinct as we like to think. If we consider the environment, then enhancements to a system may be either development or maintenance, depending on the surrounding systems.

The center of the universe is moving

The real universe, the one in which we live and has planets and solar systems and galaxies, has no center. It is "finite but unbounded" which sounds a bit strange until you realize that the surface of the Earth is also finite but unbounded. There is no edge of the Earth, no end, no boundary. Yet it has a finite size. (The Earth as a planet has a center, but the surface of the Earth does not.)

The IT universe does have centers. For decades, the center of the hardware universe has been the desktop PC and the center of the software universe has been Microsoft Windows and applications for Windows.

That is changing. Windows is no longer the software center of the IT universe. The desktop PC is no longer the hardware center of the IT universe.

The center of the IT universe for consumers has shifted to Apple and Google. The popularity of the iPad, the iPhone, and Android phones shows this. Individuals are happy to purchase these devices. PCs, in contrast, are purchased grudgingly. The purchase of a PC does not instill excitement but resentment.

The center of the IT universe for enterprises remains close to PCs and Microsoft Windows, but it too is moving to cloud computing and mobile devices. Microsoft recognizes this; it has been expanding its Azure cloud services and selling tablets and phones. While it has had little success with mobile devices, it does enjoy some with cloud services. Microsoft is supporting multiple operating systems; its Office products now run on Apple iPads and Android devices.

What does this change mean for the rest of us?

Well, for consumers it means that we will see more options. Instead of the old world of "Windows-only applications running on Microsoft Windows on desktops or laptops", we will see services on Azure available on the device of our choosing.

For enterprises, the same options will appear. This fits in with the "Bring Your Own Device" philosophy, which shifts the costs of hardware from employers to employees.

For developers, the picture is more complex. The old method of developing an application (especially an enterprise application) for Windows only (because Windows was the center of the universe) must give way to a process that develops applications for multiple platforms. The new development paradigm must be mobile/cloud with multiple cloud apps and a solid cloud design.

Microsoft is supporting this new paradigm. Azure supports non-Microsoft products such as Linux. Visual Studio supports non-Microsoft products such as Git, and now targets iOS and Android in addition to Windows.

Almost overnight, the modern Windows-only applications have been graduated to the status of legacy systems.

Software is not always soft

Tim O'Reilly asks "Can hardware really change as fast as software":

Can hardware really change as fast as software? Xiaomi thinks so http://t.co/3sjquA2XUa via @jstogdill #solidconf
— Tim O'Reilly (@timoreilly) March 24, 2014

We're used to the idea that software changes faster than hardware. It's widely accepted as common knowledge. ("Of course software changes faster than hardware! Software is soft and hardware is hard!")

Yet it's not that simple.

Software is easy to change... sometimes. There are times when software is easier to change, and there are times when software is harder to change.

Tim O'Reilly's tweet (and the referenced article) consider software in the context of cell phones. While cell phones have been changing over time, the apps for phones tend to "rev" faster. But consider the software used on PCs. Sometimes PC software changes at a rate much slower than the hardware. The "Windows XP problem" is an example: people stay with Windows XP because their software runs on Windows XP (and not later versions of Windows).

Long-term software is not limited to PCs. Corporations and governments have large systems built with mainframe technology (COBOL, batch processing) and these systems have outlasted several generations of mainframe hardware. These systems are resistant to change and do not easily translate to out current technology set of virtualized servers and cloud computing.

What makes some software easy to change and other software hard? In my view, the answer is not in the software, but in the culture and processes of the organization. "Hard" software is a result, not a cause.

Development teams that use automated testing and that refactor code frequently have a better chance of building "soft" software -- software that is easy to change. Tests keep the developers "honest" and alert them to problems. Automated tests are cheap to run and therefore run frequently, giving developers immediate feedback. Comprehensive automated tests are cheap to run and, well, comprehensive, so developers get complete feedback and alerted to any deviations from requirements.

Refactoring is important; it allows developers to improve the code over time. We rarely get code right the first time. Often we are happy that it works, and we don't care about the simplicity or the consistency of the code. Refactoring lets us re-visit that code and make it simpler and consistent -- both of which make it easier to understand and change.

Development teams that use manual methods of testing (or no testing!) have little chance at building "soft" software. Without automated tests, the risk of introducing a defect while making a change is high. Developers and managers will both avoid unnecessary changes and will consider refactoring to be an unnecessary change. The result is that code is developed but never simplified or made consistent. The code remains hard to read and difficult to change.

If you want software to be soft -- to be easy to change -- then I encourage automated testing. I see no way to get "soft" software without it.

On the other hand, if you want "hard" software -- software that is resistant to change -- then skip the automated testing. Build a culture that avoids improvements to the code and allows only those changes that are necessary to meet new requirements.

But please don't complain about the difficulty of changes.

Readable does not necessarily mean what you think it means

An easy way to start an argument among programmers (or possibly a fist-fight) is to ask about the readability of programming languages. Most developers have strong opinions about programming languages, especially when you are not paying them.

I'm not sure that "readability" is an aspect of a programming language. Instead, I think readability is an aspect of a specific program (written in any language). I have used many languages (assembly language, BASIC, FORTRAN, Pascal, COBOL, dBase, RBase, Visual Basic, Delphi, C, C++, Java, C#, Perl, Ruby, and Python, to name the popular ones), and examined many programs (small, large, tutorials, and production-level).

I've seen readable programs in all of these languages (including assembly language). I've seen hard-to-read programs in all of those languages, too.

If readability is not an aspect of a programming language but of the program itself, then what makes a program readable? Lots of people have commented on the topic over the decades. The popular ideas are usually:

• Use comments
• Follow coding standards
• Use meaningful names for variables and functions (and classes)
• Use structured programming techniques
• Have high cohesion within modules (or classes) and low coupling

These are all worthy ideas.

I would like to add one more: balance the levels of abstraction. Programs of non-trivial complexity are divided into modules. (For object-oriented programming, these divisions are classes. For structured programming, these are functions and libraries.) A readable program will have a balanced set of modules. By "balanced", I mean in terms of size and complexity.

Given a program and its modules, you can take measurements. Typical measurements include: lines of code (derided by many, and for good reasons), cyclomatic complexity, software "volume", and function points. It doesn't particularly matter which metric; any of them give you relative sizes of modules. Once you have module sizes, you can plot those sizes on a chart.

Frequently, readable programs have a smooth curve of module sizes. And just as frequently, hard-to-read programs have a jagged curve. Hard-to-read programs tend to have one or a few large modules and a few or many small modules.

I'm not sure why such a correlation exists. (And perhaps it doesn't; I admit that my observations are limited and the correlation may be proven false with a larger data set.)

Yet I have a theory.

As a programmer examines the code, he frequently moves from one module to another, following the logic and the flow of control. Modules divide the program not only into smaller components but also into levels. The different levels of the program handle different levels of abstraction. Low levels contain small units of code, small collections of data. Higher levels contain larger units of code that are composed of smaller units.

Moving from module to module means (often) moving from one level of organization to another, lower, level. With a smooth distribution of module sizes, the cognitive difference between levels is small. A path from the top level to a module on the bottom level may pass through many intermediate modules, but each transition is small and easy to understand.

A program with a "lumpy" distribution of module sizes, in contrast, lacks this set of small jumps from level to level. Instead, a program with a lumpy distribution of module sizes (and module complexity) has a lumpy set of transitions from layer to layer. Instead of a smooth set of jumps from the top level to the bottom, the jumps occur erratically, some small and some large. This disparity in jump size puts a cognitive load on the reader, making it hard to follow the changes.

If my theory is right, then a readable program is one that consists of layers, with the difference between any two adjacent layers no larger than some critical amount. (And, conversely, a hard-to-read program has multiple layer-pairs that have differences larger than that amount.)

I think that readability is an attribute of a program, not a programming language. We can write readable programs in any language (well, almost any language; a few obscure languages are designed for obfuscation). I think that the commonly accepted ideas for readable programs (comments, coding standards, meaningful names) are good ideas and helpful for building readable programs. I also think that we must structure our programs to have small differences between levels of abstraction.

Measure code complexity

We measure many things on development projects, from the cost to the time to user satisfaction. Yet we do not measure the complexity of our code.

One might find this surprising. After all, complexity of code is closely tied to quality (or so I like to believe) and also an indication of future effort (simple code is easier to change than complicated code).

The problem is not in the measurement of complexity. We have numerous techniques and tools, spanning the range from "lines of code" to function points. There are commercial tools and open source tools that measure complexity.

No, the problem is not in techniques or tools.

It is a matter of will. We don't measure complexity because, in short, we don't want to.

I can think of a few reasons that discourage the measurement of source code complexity.

- The measurement of complexity is a negative one. That is, more complexity is worse. A result of 170 is better than a result of 270, and this inverted scale is awkward. We are trained to like positive measurements, like baseball scores. (Perhaps the golf enthusiasts would see more interest if they changed their scoring system.)

- There is no direct way to connect complexity to cost. While we understand that a complicated code base is harder to maintain that a simple one, we have no way of converting that extra complexity into dollars. If we reduce our complexity from 270 to 170 (or 37 percent), do we reduce the cost of development by the same percentage? Why or why not? (I suspect that there is a lot to be learned in this area. Perhaps several Masters theses can be derived from it.)

- Not knowing the complexity shifts risk from managers to developers. In organizations with antagonistic relations between managers and developers, a willful ignorance of code complexity pushes risk onto developers. Estimates, if made by managers, will ignore complexity. Estimates made by developers may be optimistic (or pessimistic) but may be adjusted by managers. In either case, schedule delays will be the fault of the developer, not the manager.

- Developers (in shops with poor management relations) may avoid the use of any metrics, fearing that they will be used for performance evaluations.

Looking forward, I can see a time when we do measure code complexity.

- A company considering the acquisition of software (including the source code), may want an unbiased opinion of the code. They may not completely trust the seller (who is biased towards the sale) and they may not trust their own people (who may be biased against 'outside' software).

- A project team may want to identify complex areas of their code, to identify high-risk areas.

- A development team may wish to estimate the effort for maintaining code, and may include the complexity as a factor in that effort.

The tools are available.

I believe that we will, eventually, consider complexity analysis a regular part of software development. Perhaps it will start small, like the adoption of version control and automated testing. Both of those techniques were at one time considered new and unproven. Today, they are considered 'best practices'.

The Red Queen's Race requires awareness

Does your software development project use mainstream technology? (Let's assume that you care about your technology.) Some project managers want to stay in the mainstream, others want to stay ahead of the crowd, and some want leading edge tech.

Starting a project at a specific position in technology is easy. Keeping that position, on the other hand, is not so easy.

Over time, languages and compilers and libraries change. There are new versions with enhanced features and bug fixes.


Source code, once written, is a "stake in the ground". It is fixed in place, tied to a language, possibly a compiler, and probably several libraries. Keeping up with those changes requires effort. Just how much effort will vary from language to language. The C++ language has been fairly stable over its thirty-year life; Visual Basic changed dramatically in the 1990s.

Thus we have a variant of the Red Queen's Race, in which one must run just to stay in place. (In the proper race, described in "Alice Through the Looking Glass", one must run as fast as one can. I've reduced the mandate for this discussion.) A software development project must, over time, devote some effort towards "running in place", that is, keeping up with the toolset.

This effort may be small (installing a new version of the compiler) or large (a new compiler and changes to a majority of the source modules). Sometimes the effort is very large: converting a project from Perl to Python is a major effort.

Failing to move to the current tools means that you slowly drift back, and possibly fall out of your desired position. A project that starts in the leading edge drifts to the mainstream, and a project in the mainstream becomes a laggard.

The Red Queen's Race for software requires not just changes to technology (updates to compilers and such) but also an awareness of technology. In the day-to-day activities of a software project, it is easy to focus inwards, looking at new requirements and defect reports. Maintaining one's position within tech requires looking outward, at updates and new technologies and techniques. You must be aware of updates to your toolset. You must be aware of new tools for testing and collaboration. You must be aware of other groups and their technologies.

When running in a herd, it's good to look at the herd, at least once in a while.

Mobile apps are (mostly) games, and that changes the industry

If you look at the spending on mobile apps, the dollars are weighted towards games.

These figures are skewed, or at least distributed the way they are, due to the nature of users and the state of mobile apps. Mobile devices tend to be used by individuals, not corporations. Yes, corporations purchase lots of Blackberry devices, iPhones, and Android phones. But individuals purchase the larger number. Why wouldn't people purchase games for their own devices? And why would a person purchase something boring like a word processor?

This distribution may change in the future, but I'm thinking that games will retain the largest portion of mobile app revenue. If they do, it may have effects on the development industry.

The last big changes in the focus of the software development industry may have been with the PC revolution. That revolution gave us computers small enough to be used by individuals, and those computers spurred the advance of development tools: editors, compilers, debuggers, and IDEs. The IDE was a significant advance over the previous approach of separate tools.

Before PCs, computers were expensive resources that had to be shared judiciously. Applications were designed for large projects that provided return on investment: ballistics calculations, statistical analysis, accounting, inventory, and payroll. The PC revolution also lead to the rise of word processors and spreadsheets. Individuals become much more productive in the corporation. Networks and e-mail gave us additional productivity.

Throughout the PC revolution, the focus was on corporate applications. Word processors, spreadsheets, and databases are clearly in the commercial (and government) domain. Compilers and debuggers are tools for programmers, a specific industry (and a very small number of hobbyists). Yes, there were games and there were personal digital assistants. But the money was in the office.

With mobile, the money has shifted to games. One can argue that the money was shifting to games without mobile devices: Xbox, PS2, and Wii are all successful (and profitable) platforms. Perhaps the shift was occurring and mobile happened to come along at the right time.

Game applications are definitely not for corporations. Games are for individuals, or groups of game-playing individuals.

With the shift of money from office applications to games, how will the development industry change? When people are the primary client, and games the primary type of application, will the top developers  focus their effort on game development? Will non-game corporations struggle to hire developers, or get only the second-tier programmers?

Will a games-driven software industry resemble Hollywood and the movie industry? Games and movies are both entertainment, not productivity tools. Some movies have games. (And some games now have movies.)

The Cheapening of IT

The prices for computing equipment, over the years, have moved in one direction: down. I believe that the decrease in prices for hardware has an affect on our willingness to pay for software.

In the early 1960s, a memory expansion for the IBM 1401 provided 8K of what we today call RAM, at a price of $258,000. That was only the expansion pack of memory; the entire system cost several times that amount. With an investment of over a million dollars for hardware, an additional investment of several tens of thousands of dollars for software was quite the bargain.

In 1977, a Heathkit 8-bit microcomputer with an 8080 processor, 4K of RAM, and a cassette tape recorder/player (used for long-term storage prior to floppy disks), cost almost $1500. Software for such a computer ran from $20 (for a simple text editor) to $400 (for the Microsoft COBOL compiler).

Today, smart phone or tablet costs range from $200 to $1000. (Significantly less than the Heathkit 8-bit system, once you account for inflation.) Tablet apps can cost as much as $10. Some are more, and some are free.

What affect does this decrease in the hardware cost have on the cost of software?

Here's my theory: as the cost of hardware decreases, the amount that we are willing to pay for software also decreases. I can justify spending $400 for software when the hardware costs several times that amount. But I have a harder time spending $400 on software when the hardware costs less than that. My bias is for hardware, and I am assigning higher intrinsic value to the hardware than the software. (The reasons behind this are varied, from the physical nature of hardware to the relationship with the vendor. I'm pretty sure that one could find a Master's thesis in this line of study.)

But if a cheapening of the hardware leads to a cheapening of the software, how does that change the industry? Assuming that the theory is true, we should see downward pressure on the cost of applications. And I think that we have seen this. The typical phone and tablet app holds a retail price that is significantly less than the price for a typical desktop PC application. "Angry Birds" costs only a fraction of the price of Microsoft Office.

I expect that this cost bias will extend to PC apps that move to tablets. Microsoft Word on the Surface will be priced at under $40 (perhaps as an annual subscription) and possibly less. The initial release of the Surface includes a copy of Word, although it is restricted to non-commercial use.

I also expect that the price of desktop PC apps will fall, keeping close to the prices of tablet apps. Why spend $400 for Word on the PC when one can get it for $40 on the tablet? The reduced price of apps on one platform drives down the price of apps on all platforms.

The cheapening affect may go beyond off-the-shelf PC applications. As the prices of desktop applications fall, we may see pressure to reduce the price of server-based systems, or server components of multiplatform systems. Again, this will be driven not by technology but by psychology: I cannot justify a multi-thousand dollar cost for a server component when the corresponding desktop applications have low costs. The reduced prices of desktop applications drives down the prices of equivalent server applications. Not all server applications, mind you; only the server applications that have desktop equivalents, and only then when those desktop equivalents are reduced in price to match tablet apps.

The general reduction of prices for desktop and server applications may create difficulties for the big consulting shops. These shops charge high prices for the development of custom applications for businesses. Psychology may cause headaches for their sales teams: why should I spend hundreds of thousands of dollars on a custom app (which includes clients for desktop PCs, tablets, and smartphones, of course) when I can see that powerful, competent apps are marketed for less than $10 per user? While there is value is a custom application, and while a large company may need many "downloads" for their many users, the argument for such high prices becomes difficult. Is a custom app really adding that much value?

Look for the large consulting houses to move into new technologies such as cloud and "big data" as ways of keeping their rates high. By selling these new technologies, the consulting houses can offer something that is not readily apparent in the off-the-shelf apps. (At least until their customers figure out that the off-the-shelf apps are also using cloud and "big data" tech.)

All of this leads to downward pressure on the prices of apps, whether they are simple games or complex systems. That pressure, in turn, will put downward pressure on development costs and upward pressure for productivity. Where a project was run with a project manager, three tech leads, ten developers, three testers, two analysts, and a technical writer, future projects may be run with a significantly smaller team. Perhaps the team will consist of one project manager, one tech lead, three developers, and one analyst. I'm afraid the "do more with less" exhortation will be with us for a while.

$1000 per hour

Let's imagine that you are a manager of a development team. You hire (and fire) members of the team, set goals, review performance, and negotiate deliverables with your fellow managers.

Now let's imagine that the cost of developers is significantly higher than it is today. Instead of paying the $50,000 to $120,000 per year, you must pay $1000 per hour, or $2,000,000 per year. (That's two million dollars per year.) Let's also imagine that you cannot reduce this cost through outsourcing or internships.

What would you do?

Here is what I would do:

• I would pick the best of my developers and fire the others. A smaller team of top-notch developers is  more productive than a large team of mediocre developers.
• I would provide my developers with tools and procedures to let them be the most productive. I would weigh the cost of development tools against the time that they would save.
• I would use automated testing as much as possible, to reduce the time developers spend on manual testing. If possible, I would automate all testing.
• I would provide books, web resources, online training, and conferences to the developers, to give them the best information and techniques on programming.

In other words, I would do everything in my power to make them productive. When their time costs money, saving their time saves me money. Sensible, right?

But the current situation is the same. Developers cost me money. Saving their time saves me money.

So why aren't you doing what you can to save them time?

... or the few

Software development is maturing.

I suppose that is has been since the Eldar days, when programming meant wiring plug-boards. Programming languages and tools have been changing since the first symbolic assemblers.

There have been two types of changes ("improvements"?) for programming: changes that benefit the individual programmer and changes that benefit the team.

Changes that benefit the individual include:

 - FORTRAN
 - Unix, which let a programmer use many tools and even build his own
 - Integrated development environments
 - Interactive debuggers
 - Faster processors and increased memory
 - Unit tests (especially automated unit tests)
 - Diagnostic programs such as 'lint'
 - Managed environments with garbage collection (eliminating the need for 'delete' operations)
 - Early forms of version control systems

Changes that benefit the team include:

 - COBOL, with its separation of concerns and support for discrete modules
 - Early operating systems that scheduled jobs and managed common resources
 - The 'patch' program, which allowed for updates without transmitting the entire source
 - Network connections (especially internet connections)
 - Code reviews
 - Distributed version control systems
 - github

One can argue that any of these technologies help individuals and teams. A better debugger may directly help the programmer, yet it helps the team by making the developer more productive. Some technologies are better for the individual, and others better for the team. But that's not the point.

The point is that we, as an industry, look to improve performance for individuals and teams, not just individuals. We look at teams as more than a collection of individuals. If we considered teams nothing more than a group of people working on the same project, we would focus our efforts on individual performance and nothing else. We would develop better tools for individuals, and not for teams.

Some languages are designed for teams, or at least have aspects designed for teams. The Python and Go languages have strong ideas about code style; Python enforces rules for indentation and Go rules for bracket placement. They are not the first; Visual Basic would auto-space and auto-capitalize your source code. That may have been a function of the editor, but since the editor and interpreter were distributed together one can consider the action part of the language.

Python's indentation rules, Go's bracket rules, and Visual Basic's auto-capitalization are all beneficial to the individual programmer, but they are more beneficial to the team. They enforce style upon the source code. Such enforced style ensures that all members of the team (indeed, all members of those programming communities) use the same style. Programmers can more easily move from one project to another, and from code contributed by one person to another. Some programming teams () using other languages) enforce local styles, but these languages have it built into their DNA.

Enforced style is a move against the "cowboy coder", the rebel programmer who insists on doing things "his way". Programming languages are not a democracies -- team members don't get to vote on the rules of the language that they are using -- but it is a step towards government.

Let's count that as maturation.

Build on clouds

Folks have put effort (some folks lots of effort) into making the cloud look and behave like plain old PC applications. Not just the look and feel of apps, but the techniques to build those apps. Microsoft has designed their tools to make the development of cloud apps a lot like the development of PC applications. (Their tools also make the development of web apps similar to the development of PC apps.)

I think that this is the wrong approach.

I recognize the benefits of similar environments and tools. A single method is easier to learn and support. One can move from development on one project to development on another (different platform) project.

But there is nothing in the PC app development model that makes it the right model for all platforms.

The PC app toolset has a long history. We started with the tools and techniques of the mainframe and minicomputer platforms, but didn't keep them. We adopted the notions of compilers, interactive command lines, and version control, and then built our own tools that leveraged PC resources: the IDE, interactive debuggers, and GUI designers. We abandoned the processes of desk-checking software and symbol cross-references; we built modern processes, from code reviews to automated testing.

With cloud computing, expect new methods and tools that leverage the resources of the cloud. You won't use the new techniques if you stay in the PC mindset.

If you must have a single development platform, make it the cloud. That is, make the process and tools for the development of PC apps look and feel like the process for the development of cloud apps. Instead of adapting the cloud to the PC app process, create a process for cloud apps and then adapt it to the PC app process.

It's a poor atom blaster that doesn't scale

In Isaac Asimov's "Foundation" novel, the character Salvor Hardin utters the phrase "It's a poor atom blaster that doesn't point both ways.". The same can be said of "scaling" the process of adjusting an application or system to handle a different workload.

Most people think of scaling in the upward direction. Having built a prototype or proof-of-concept system, they ask themselves "will this design 'scale'?", meaning "will this design hold up under a large workload?". Frequently the system under discussion is a database, but it can be a social network or just about anything.

Yet scaling works in two directions. One can create a system that works well for a single user but fails with many users, and one can create a system that works for many users but fails for a small number. And the system does not have to be a database or even a system at all.

One example is Microsoft's products for version control. Let's consider the Visual SourceSafe and Team Foundation Server products. The former is (was, as it is no longer sold) usable by small teams, perhaps up to twenty people. Beyond that, it is hard to manage user rights and the underlying database. The latter (Team Foundation Server) on the other hand is built for large teams -- but is unsuitable for a small team of less than five. TFS has a fairly heavy administration cost, and it sucks up time and energy. A large team can afford to dedicate a person or two to the administration; a small team cannot.

Languages are also affected by scaling. Microsoft's Visual Basic languages (prior to VB.NET) were very good for small Windows applications. Simple applications could be made quite easily, but complex applications were harder. Visual Basic had some object-oriented constructs, but it was mostly procedural and one needed a lot of discipline to create a middling-complex application. Also, Visual Basic had some behaviors that limited the complexity of applications, such as its inability to display modal dialogs within modal dialogs.

Microsoft's Visual C++, in contrast to Visual Basic, was better suited for complex applications. It had full object-oriented constructs. It allowed any number of dialogs (modal or non-modal). But it had a high start-up cost. A single person could use it, but it was better for teams.

Scaling down is important for languages. Developers need tools to perform all sorts of tasks, and some applications are sensible for single users. Microsoft's tools are designed for large problems with large solutions implemented by large teams. Java is in a better situation, with language hacks like Scala and Groovy.

The current crop of scripting languages (Perl, Python, and Ruby) scale well, allowing single developers, small teams, and large teams. (Perl perhaps not quite so easily with large teams.)

With the rise of smartphones, apps have to move to smaller screens and virtual keyboards. (And virtual keyboards make a difference -- one does not type a term paper on a cell phone.) The assumptions of large screen and full keyboards do not apply in the cell phone arena.

Scaling down is just as important as scaling up. Scaling applies to users, data, and developers. Not every application, system, and development effort must scale from smartphone to cloud, but the system designers should look both ways before choosing their strategies.