Holding back time

An old episode of the science-fiction series "Dr. Who", the villain roams the galaxy and captures entire planets, all to power "time dams" used to prevent his tribal matriarch from dying. The efforts were in vain, because while one can delay changes one cannot hold them back indefinitely.

A lot of effort in IT is also spent on "keeping time still" or preventing changes.

Projects using a "waterfall" process prevent changes by agreeing early on to the requirements, and then "freezing" the requirements. A year-long project can start with two-month phase to gather, review, and finalize requirements; the remainder of the year is devoted to implementing those requirements, exactly as agreed, with no changes or additions. The result is often disappointing. Delivered systems were incorrect (because the requirements, despite review, were incorrect) or incomplete (for the same reason) and even if neither of those were true the requirements were a year out of date. Time had progressed, and changes had occurred, outside of the project "bubble".

Some waterfall-managed projects allow for changes, usually with an onerous "change control" process that requires a description and justification of the change and agreement (again) among all of the concerned parties. This allows for changes, but puts a "brake" on them, limiting the number and scope of changes.

But project management methodologies are not the only way we try to hold back time. Other areas that we try to prevent changes include:

Python's "requirements.txt" file, which lists the required packages. When used responsibly, it lists the required packages and the minimum version of each package. (A good idea as one does need to know the packages and the versions, and this was a consistent method.) Some projects try to hold back changes by specifying an exact version of a package (such as "must be version 1.4 and no other") in fear that a later version may break something.

Locking components to specific versions will eventually fail: a component will not be available, or the specified version will not work on a new operating system or in a new version of the interpreter. (Perhaps even the Python interpreter itself, if held back in this manner, will fail.)

Containers, which contain the "things that an application needs". Many "containerized" applications contain a database and the database software, but they can also include other utilities. The container holds a frozen set of applications and libraries, installed each time the container is deployed. While they can be updated that doesn't mean they are updated.

Those utilities and libraries that are "frozen in time" will eventually cause problems. They are not stand-alone; they often rely on other utilities and libraries, which may not be present in the container. At some point, the "outside" libraries will not work for the "inside" applications.

Virtual machines to run old versions of operating systems, to run old versions of applications that run only on old versions of operating systems. Virtual machines can be used for other purposes, and this is yet another form of "holding back time".

Virtual machines with old versions of operating systems, running old versions of applications, also have problems. Their ability to communicate with other systems on the network will (probably) break, due to expired certificates or a change in a protocol.

All of these techniques pretend to solve a problem. But they are not really solutions -- they simply delay the problem. Eventually, you will have an incompatibility, somewhere. But that isn't the biggest problem.

The biggest problem may be in thinking that you don't have a problem.

Cloud, like other forms of computers, changes over time

Cloud computing has been with us a while. In its short life, and like other types of computing, it has changed.

"Cloud" started out as the outsourcing of system administration.

Then "cloud" was about scalability, and the ability to "spin up" servers as you needed them and "spin down" servers when they were not needed.

Shortly after, "cloud" was a cost-control measure: pay for only the servers you use.

For a while, "cloud" was a new type of system architecture with dedicated servers (web, database) connected by message queues.

Then "cloud" was about microservices, which are small web services that are less than complete applications. (Connect the right microservices in the right way, and you have an application!)

Lately, "cloud" has been all about containers, and the rapid and lightweight deployment of applications.

So what is "cloud computing", really?

Well, it's all of these things. As I see it, cloud computing is a new form of computing, difference from mainframe computing, desktop computing, and web applications. As a new form of computing, it has taken us a while to fully understand it.

We had similar transitions with desktop (or PC) computing and web applications. Early desktop microcomputers (the Apple II, the TRS-80, and even the IBM PC) were small, slow, and difficult to use. Over time, we modified those PCs: powerful processors, bigger displays, more memory, simpler attachments (USB instead of serial), and better interfaces (Windows instead of DOS).

Web applications went through their own transitions, from static web pages to CGI Perl scripts to AJAX applications to new standards for HTML.

Cloud computing is undergoing a similar process. It shouldn't be a surprise; this process of gradual improvement is less about technology and more about human creativity. We're always looking for new ways of doing things.

One can argue that PCs and web applications have not stopped changing. We've just added touchscreens to desktop and laptop computers, and we've invented NoSQL databases for web applications (and mobile applications). It may be that cloud computing will continue to change, too.

It seems we're pretty good at changing things.

Big changes happen early

Software has a life cycle: It is born, it grows, and finally dies. That's not news, or interesting. What is interesting is that the big changes in software happen early in its life.

Let's review some software: PC-DOS, Windows, and Visual Basic.

PC-DOS saw several versions, from 1.0 to 6.0. There were intermediate versions, such as versions 3.3 and 3.31, so there were more that six versions.

Yet the big changes happened early. The transition from 1.0 to 2.0 saw big changes in the API, allowing new device types and especially subdirectories. Moving from version 1.0 to 2.0 required almost a complete re-write of an application. Moving applications from version 2.0 to later versions required changes, but not as significant. The big changes happened early.

Windows followed a similar path. Moving from Windows 1 to Windows 2 was a big deal, as was moving from Windows 2 to Windows 3. The transition from Windows 3 to Windows NT was big, as was the change from Windows 3.1 to Windows 95, but later changes were small. The big changes happened early.

Visual Basic versions 1, 2, and 3 all saw significant changes. Visual Basic 4 had some changes but not as many, and Visual Basic 5 and 6 were milder. The big changes happened early. (The change from VB6 to VB.NET was large, but that was a change to another underlying platform.)

There are other examples, such as Microsoft Word, Internet Explorer, and Visual Studio. The effect is not limited to Microsoft. Lotus 1-2-3 followed a similar arc, as did dBase, R:Base, the Android operating system, and Linux.

Why do big changes happen early? Why do the big jumps in progress occur early in a product's life?

I have two ideas.

One possibility is that the makers and users of an application have a target in mind, a "perfect form" of the application, and each generation of the product moves closer to that ideal form. The first version is a first attempt, and successive versions improve upon previous versions. Over the life of the application, each version moves closer to the ideal.

Another possibility is that changes to an application are constrained by the size of the user population. A product with few users can see large changes; a product with many users can tolerate only minor changes.

Both of these ideas explain the effect, yet they both have problems. The former assumes that the developers (and the users) know the ideal form and can move towards it, albeit in imperfect steps (because one never arrives at the perfect form). My experience in software development allows me to state that most development teams (if not all) are not aware of the ideal form of an application. They may think that the first version, or the current version, or the next version is the "perfect" one, but they rarely have a vision of some far-off version that is ideal.

The latter has the problem of evidence. While many applications grow there user base over time and also shrink their changes over time, not all do. Two examples are Facebook and Twitter. Both have grown (to large user bases) and both have seen significant changes.

A third possibility, one that seems less theoretical and more mundane, is that as an application grows, and its code base grows, it is harder to make changes. A small version 1 application can be changed a lot for version 2. A large version 10 application has oodles of code and oodles of connected bits of code; changing any bit can cause lots of things to break. In that situation, each change must be reviewed carefully and tested thoroughly, and those efforts take time. Thus, the older the application, the larger the code base and the slower the changes.

That may explain the effect.

Some teams go to great lengths to keep their code well-organized, which allows for easier changes. Development teams that use Agile methods will re-factor code when it becomes "messy" and reduce the couplings between components. Cleaner code allows for bigger and faster changes.

If changes are constrained not by large code but by messy code, then as more development teams use Agile methods (and keep their code clean) we will see more products with large changes not only early in the product's life but through the product's life.

Let's see what happens with cloud-based applications. These are distributed by nature, so there is already an incentive for smaller, independent modules. Cloud computing is also younger than Agile development, so all cloud-based systems could have been (I stress the "could") developed with Agile methods. It is likely that some were not, but it is also likely that many were -- more than desktop applications or web applications.

Apple wins the race for laptops -- now what?

Apple is successful, in part, due to its hardware design. Its products are lovely to look at and comfortable to hold. Its portable devices are thinner and lighter than the competition. Apple has defined the concept of laptop for the last several years -- perhaps the last decade. Apple has set the standard and other manufacturers have followed.

The latest MacBook design is light, sleek, and capable. It is the ultimate in laptop design. And by "ultimate", I mean "ultimate". Apple has won the laptop competition. And now, they have a challenge: what to do next?

Apple's advantage of the MacBook will last for only a short time. Already other manufacturers are creating laptops that are just as thin and sleek as the MacBook. But Apple cannot develop a thinner, sleeker MacBook. The problem is that there are limits to physical size. There are certain things that make a laptop a laptop. You need a display screen and a keyboard, along with processor, memory, and I/O ports. While the processor, memory, and I/O ports can be reduced in size (or in number), the screen and keyboard must be a certain size, due to human physiology.

So how will Apple maintain its lead in the laptop market?

They can use faster processors, more memory, and higher resolution displays.

They can add features to Mac OS X.

They can attempt thinner and sleeker designs.

They may add hardware features, such as wireless charging and 3G/4G connections.

But there is only so much room left for improvement.

I think the laptop market is about to change. I think that laptops have gotten good enough -- and that there are lots of improvements in other markets. I expect manufacturers will look for improvements in tablets, phones, and cloud technologies.

Such a change is not unprecedented. It happened in the desktop PC market. After the initial IBM PC was released, and after Compaq made the first clone, desktop PCs underwent an evolution that lead to the PC units we have today -- and have had for the past decade. Desktop PCs became "good enough" and manufacturers moved on to other markets.

Now that laptops have become good enough, look for the design of laptops to stabilize, the market to fragment among several manufacturers with no leader, prices to fall, and innovation to occur in other markets.

Which doesn't mean that laptops will become unimportant. Just as PCs had a long life after they became "good enough" and their design stabilized, laptops will have a long life. They will be an important part of the IT world.

The cloud brings change

Perhaps the greatest effect that cloud computing has on operations is change -- or more specifically, a move away from the strict policies against change.

Before cloud technology, operations viewed hardware and software as things that should change rarely and under strictly controlled conditions. Hardware was upgraded only when needed, and only after long planning sessions to review the new equipment and ensure it would work. Software was updated only during specified "downtime windows", typically early in the morning on a weekend when demand would be low.

The philosophies of "change only when necessary" and "only when it won't affect users" were driven by the view of hardware and software. Before cloud computing, most people had a mindset that I call the "mainframe model".

In this "mainframe model," there is one and only one computer. In the early days of computing, this was indeed the case. Turning the computer off to install new hardware meant that no one could use it. Since the entire company ran on it, a hardware upgrade (or a new operating system) meant that the entire company had to "do without". Therefore, updates had to be scheduled to minimize their effect and they had to be carefully planned to ensure that everything would work upon completion.

Later systems, especially web systems, used multiple web servers and often multiple data centers with failover, but people kept the mainframe model in their head. They carefully scheduled changes to avoid affecting users, and they carefully planned updates to ensure everything would work.

Cloud computing changes that. With cloud computing, there is not a single computer. There is not a single data center (if you're building your system properly). By definition, computers (usually virtualized) can be "spun up" on demand. Taking one server offline does not affect your customers; if demand increases you simply create another server to handle the additional workload.

The ability to take servers offline means that you can relax the scheduling of upgrades. You do not need to wait until the wee hours of the morning. You do not need to upgrade all of your servers at once. (Although running servers with multiple versions of your software can cause other problems. More on that later.)

Netflix has taken the idea further, creating tools that deliberately break individual servers. By doing so, Netflix can examine the failures and design a more robust system. Once a single server fails, other servers take over the work -- or so Netflix would like. If servers don't pick up the workload, Netflix has a problem and changes their code.

Cloud technology lets us use a new model, what I call the "cloud model". This model allows for failures at any time -- not just from servers failing but from servers being taken offline for upgrades or maintenance. Those upgrades could be hardware, virtualized hardware, operating systems, database schemas, or application software.

The cloud model allows for change. It requires a different set of rules. Instead of scheduling all changes for a single downtime window, it distributes changes over time. It mandates that newer versions of applications and databases work with older versions, which probably means smaller, more incremental changes. It also encourages (perhaps requires) software to administer the changes and ensure that all servers get the changes. Instead of growing a single tree you are growing a forest.

The fact that cloud technology brings change, or changes our ideas of changes, should not be a surprise. Other technologies and techniques (agile development, dynamic languages) have been moving in the same direction. Even Microsoft and Apple are releasing products more quickly. Change is upon us, whether we want it or not.

With no clear IT leader, expect lots of changes

The introduction of the IBM PC was market-wrenching. Overnight, the small, rough-and-tumble market of microcomputers with diverse designs from various small vendors became large and centered around the PC standard.

From 1981 to 1987, IBM was the technology leader. IBM lead in sales and also defined the computing platform.

IBM's leadership fell to Compaq in 1987, when IBM introduced the PS/2 line with its new (incompatible) hardware. Compaq delivered old-style PCs with a faster buss (the EISA buss) and notably the Intel 80386 processor. (IBM stayed with the older 80286 and 8086 processors, eventually consenting to provide 80386-based PS/2 units.) Compaq even worked with Microsoft to deliver newer versions of MS-DOS that recognized larger memory capacity and optical disc readers.

But Compaq did not remain the leader. It's leadership declined gradually, to the clone makers and especially Dell, HP, and Gateway.

The mantle of leadership moved from a PC manufacturer to the Microsoft-Intel duopoly. The popularity of Windows, along with marketing skill and software development prowess led to a stable configuration for Microsoft and Intel. Together, they out-competed IBM's OS/2, Motorola's 68000 processor, DEC's Alpha processor, and Apple's Macintosh line.

That configuration held for two decades, roughly from 1990 to 2010, when Apple introduced the iPhone. The genius move was not the iPhone hardware, but the App Store and iTunes, which let one easily find and install apps on your phone (and pay for them).

Now Microsoft and Apple have the same problem: after years of competing in a well-defined market (the corporate PC market) they struggle to move into the world of mobile computing. Microsoft's attempts at mobile devices (Zune, Kin, Surface RT) have flopped. Intel is desperately attempting to design and build processors that are suitable for low-power devices.

I don't expect either Microsoft or Intel to disappear. (At least not for several years, possibly decades.) The PC market is strong, and Intel can sell a lot of its traditional (heat radiator that happen to compute data) processors. Microsoft is a competent player in the cloud arena with its Azure services.

But I will make an observation: for the first time in the PC era, we find that there is no clear leader for technology. The last time we were leaderless was prior to the IBM PC, in the "microcomputer era" of Radio Shack TRS-80 and Apple II computers. Back then, the market was fractured and tribal. Hardware ruled, and your choice of hardware defined your tribe. Apple owners were in the Apple tribe, using Apple-specific software and exchanging data on Apple-specific floppy disks. Radio Shack owners were in the Radio Shack tribe, using software specific to the TRS-80 computers and exchanging data on TRS-80 diskettes. Exchanging data between tribes was one of the advanced arts, and changing tribes was extremely difficult.

There were some efforts to unify computing: CP/M was the most significant. Built by Digital Research (a software company with no interest in hardware), CP/M ran on many different configurations. Yet even that effort could not span the differences in processors, memory layout, and video configurations.

Today we see tribes forming around multiple architectures. For cloud computing, we have Amazon.com's AWS, Microsoft's Azure, Google's App Engine. With virtualization we see VMware, Oracle's VirtualBox, the aforementioned cloud providers, and newcomer Docker as a rough analog of CP/M. Mobile computing sees Apple's iOS, Google's Android, and Microsoft's Windows RT as a (very) distant third.

With no clear leader and no clear standard, I expect each vendor to enhance their offerings and also attempt to lock in customers with proprietary features. In the mobile space, Apple's Swift and Microsoft's C# are both proprietary languages. Google's choice of Java puts them (possibly) at odds with Oracle -- although Oracle seems to be focussed on databases, servers, and cloud offerings, so there is no direct conflict. Things are a bit more collegial in the cloud space, with vendors supporting OpenStack and Docker. But I still expect proprietary enhancements, perhaps in the form of add-ons.

All of this means that the technology world is headed for change. Not just change from desktop PC to mobile/cloud, but changes in mobile/cloud. The competition from vendors will lead to enhancements and changes, possibly significant changes, in cloud computing and mobile platforms. The mobile/cloud platform will be a moving target, with revisions as each vendor attempts to out-do the others.

Those changes mean risk. As platforms change, applications and systems may break or fail in unexpected ways. New features may offer better ways of addressing problems and the temptation to use those new features will be great. Yet re-designing a system to take advantage of new infrastructure features may mean that other work -- such as new business features -- waits for resources.

One cannot ignore mobile/cloud computing. (Well, I suppose one can, but that is probably foolish.) But one cannot, with today's market, depend on a stable platform with slow, predictable changes like we had with Microsoft Windows.

With such an environment, what should one do?

My recommendations:

Build systems of small components  This is the Unix mindset, with small tools to perform specific tasks. Avoid large, monolithic systems.

Use standard interfaces  Use web services (either SOAP or REST) to connect components into larger systems. Use JSON and Unicode to exchange data, not proprietary formats.

Hedge your bets  Gain experience in at least two cloud platforms and two mobile platforms. Resist the temptation of "corporate standards". Standards are good with a predictable technology base. The current base is not predictable, and placing your eggs in one vendor's basket is risky.

Change your position  After a period of use, examine your systems, your tools, and your talent. Change vendors -- not for everything, but for small components. (You did build your system from small, connected components, right?) Migrate some components to another vendor; learn the process and the difficulties. You'll want to know them when you are forced to move to a different vendor.

Many folks involved in IT have been living in the "golden age" of a stable PC platform. They may have weathered the change from desktop to web -- which saw a brief period of uncertainty. More than likely, they think that the stable world is the norm. All that is fine -- except we're not in the normal world with mobile/cloud. Be prepared for change.

Mobile changes how we think about computing

The rise of mobile computers, while not a revolution, does introduce a significant change in our thinking of computing. I believe that this change generates angst for many.

The history of computing has seen three major waves of technology. Each of these waves has had a specific mindset, a specific way that we view computing.

Mainframes The first wave of computing was the mainframe era. Computers were large, expensive, magical boxes that were contained in sealed rooms (temples?) and attended by technicians (priests?). The main tasks of computers was to calculate numbers for the company (or government) and most jobs were either accounting or specific mathematical calculations (think "ballistics tables").

Minicomputers The second wave of computing was the minicomputer era. Computers were the size of refrigerators or washing machines and could be purchased by departments within a company (or a university). They did not need a sealed room with special air conditioning, although they were usually stored in locked rooms to prevent someone from wheeling them away. The main tasks were still corporate accounting, inventory management, order processing, and specific mathematical calculations.

Personal computers The third wave of computing saw a major shift in our mindset of computing. Personal computers could be purchased (and run) by individuals. They could be used at home or in the office (if you carried it in yourself). The mindset for personal computing was very different from the corporate-centered computing of the previous eras. Personal computing could be used for ... anything. The composition and printing of documents was handled by word processors. Spreadsheets let us calculate our own budgets. Small databases (and later larger databases) let us store our own transaction data. If off-the-shelf software was not suitable to the task, we could write our own programs.

The mindset of personal computing has been with us for over thirty years. The size and shape of personal computers has been roughly the same: the same CPU box, the same keyboard, the same monitor. We know what a PC looks like, The software has seen one major change, from DOS to Windows, but Windows has been with us for the past twenty years. We know what programs look like.

The introduction of tablets has caused us to re-think our ideas of computing. And we're not that good at re-thinking. We see tablets and phones and they seem strange to us. The size and shape are different (and therefore "wrong"); the user interface is different (and therefore "wrong"); the way we purchase applications is different (and therefore "wrong"); even the way we call applications ("apps") is different (and therefore... you get the idea).

I observe that mobile devices caused little discomfort while they remained in the consumer market. Phones that could play music and games were not a problem. Tablets that let one scroll through Facebook or read books were not a problem. These were extensions to our existing technology.

Now phones and tablets are moving into the commercial sphere, and their application is not obvious. It is clear that they are not personal computers -- their size and shape prove that. But there are more differences that cause uncertainty.

Touch interface The user interface for phones and tablets is not about keyboards and mice but about taps and swipes.

Small screen Tablets have small-ish screens, and phones have tiny screens. How can anyone work on those?

Different operating systems Personal computers run Windows (except for a few in the marketing groups that use Mac OS). Tablets run something called "Android" or something else called "iOS".

Something other than Microsoft Microsoft's entries in the phone and tablet market are not the market leaders and their operating systems have not been accepted widely.

Even Microsoft isn't Microsoft-ish Microsoft's operating system for phones and tablets isn't really Windows, it is this thing called "Windows 8". The user interface looks completely different. Windows RT doesn't run "classic" Windows programs at all (except for Microsoft Office).

The changes coming from mobile are only one front; changes to the PC are also coming.

The typical PC is shrinking Display screens have become flat. The CPU box is shrinking, losing the space for expansion cards and empty disk bays. Apple's Mac Mini, Intel's New Unit of Computing, and other devices are changing how we look at computers.

Windows is changing Windows 8 is very different from "good old Windows". (My view is that Windows 8's tiles are simply a bigger, better "Start" menu, but many disagree.)

These changes mean that one cannot stay put with Windows. You either advance into the mobile world or you advance into the new Windows world.

The brave new worlds of mobile  and Windows look and feel very different from the old world of computing. Many of our familiar techniques are replaced with something new (and strange).

We thought we knew what computers were and what computing was. Mobile changes those ideas. After thirty years of a (roughly) constant notion of personal computing, many people are not ready for a change.

I suspect that the people who are hardest hit by the changes of mobile are those aged 25 to 45; old enough to know PCs quite well but not old enough to remember the pre-PC days. This group never had to go through a significant change in technology. Their world is changing and few are prepared for the shock.

The under-25 crowd will be fine with tablets and computers. It's what they know and want.

Interestingly, the over-45 folks will probably weather the change. They have already experienced a change in computing, either from mainframes or minicomputers to personal computers, or from nothing to personal computers.

Projects must move and grow

Software development projects are like people, in that they grow over time. Technology changes around us, and we must allow for changes to our projects.

The things that change are:

• Tools, such as editors and compilers
• Processes, such as design reviews and code walk-throughs
• Source code

Tools evolve over time. New versions of compilers are released. Sometimes a new version will break existing code. The C++ standards committee works very hard to prevent such breakages. (Although they have made some changes that broke code, after long deliberations.)

A project that practices good hygiene will upgrade tools. The change does not have to be immediate, but it is within a reasonable period of time. (Perhaps six months.)

The "best practices" for software development change over time. Often these changes are made possible with the invention of tools or the release of a new product. Past changes have included the use of version control, the use of lint utilities, code reviews, and automated testing. None of these were available (cheaply) in the 1990s, and they are today. Version control has undergone several generations, from the early PVCS and CVS systems to SourceSafe and Subversion and today's TFS and 'git'.

The source code changes over time, too. Not just for the addition of new features, but improvements to the code. Programming techniques, like tools and best practices, change over time. We moved from procedural programming to object-oriented programming. We've developed patterns such as "Model View Controller" and "Model View ViewModel" which help organize our code and reduce complexity.

Changes for tools, techniques, and source code take time and effort. They must be planned and incorporated into releases. They entail risk; any change can introduce a defect. To make matters worse, such changes are "internal" and offer no direct benefit to the users. The changes are for the benefit of the development team.

I have seen a number of projects start with the then-current set of tools and techniques, only to become established and stay with those tools and techniques. The once-modern project ages into a legacy effort. It is a trap that is all too easy: the demand for new features and bug fixes overwhelm the team and there is no time for non-revenue improvements.

The "no return on investment" argument is difficult to counter. Given finite resources and the choice between a feature that provides revenue against a change that provides no revenue, it is sensible to go with the revenue feature.

Without these internal changes, the project cost rises. The increases are caused by two factors: code complexity and ability to hire staff.

Over time, changes (especially rushed changes) increase the complexity of the code and changes become much more difficult. The code, once neat and organized, becomes messy. Features are added quickly, with compromises made to quality for delivery time. Each additional change adds to the "mess" of the code.

The world of software development advances, but the project remains stuck in its original era. The tools, initially the most modern, age. They become yesterday's tools and techniques.

Another problem is staffing. Few developers are willing to work on a project that has hard-to-maintain code, old tools, and outdated processes. The few that are will do so only at elevated rates. This increases the cost of future maintenance.

Allocating time and effort (and perhaps money) to keep the project up to date is not easy. The payoff is in the long term. A good project manager balances the short-term needs and the long-term goals.