Eggs and baskets

PCWorld, a venerable trade publication-now-website of the IT realm, recently lost its YouTube video channel. The channel was disabled (or suspended? or deleted?) and no content was available. For more than eight days.

From what I can discern, IDG's YouTube account was controlled by an IDG e-mail address. Everything worked until IDG was purchased by Foundry, and Foundry changed all of IDG's e-mail addresses to Foundry addresses, didn't change the account at YouTube, and YouTube, seeing no activity on the IDG e-mail address or maybe getting bounce messages, cancelled the account.

Thus, the PCWorld video channel was unavailable for over a week.

Why didn't PCWorld restore its channel? Or make its content available on another service? 

My guess is that IDG stored all of their video content on YouTube. That is, the only copy was on YouTube. IDG probably relied on YouTube to keep backup copies and multiple servers for disaster recovery. In short, IDG followed the pattern for cloud-based computing.

The one disaster for which IDG failed to prepare was the account cancellation.

I must say here that a lot of this is speculation on my part. I don't work for PCWorld, or at IDG (um, Foundry) or at YouTube. I don't know that the sequence I have outlined is what actually happened.

My point is not to identify exactly what happened.

My point is this: cloud solutions, like any other type of technology, can be fragile. They can be fragile in ways that we do not expect.

The past half-century of computing has shown us that computers fail. They fail in many ways, from physical problems to programming errors to configuration mistakes. Those failures often cause problems with data, sometimes deleting all of it, sometimes deleting part of it, and sometimes modifying (incorrectly) part of the data. We have a lot of experience with failures, and we have built a set of good practices to recover from those failures.

Cloud-based solutions do not eliminate the need for those precautions. While cloud-based solutions offer protection against some problems, they introduce new problems.

Such as account cancellation.

Businesses (and people, often), when entering into agreements, look for some measure of security. Businesses want to know that the companies they pick to be suppliers will be around for some time. They avoid "fly by night" operations.

A risk in cloud-based solutions is account closure. The risk is not that Google (or Oracle) will go out of business, leaving you stranded. The risk is that the cloud supplier will simply stop doing business with you.

I have seen multiple stories about people or businesses who have had their accounts closed, usually for violating the terms of service. When said people or businesses reach out to the cloud provider (a difficult task in itself, as they don't provide phone support) the cloud provider refuses to discuss the issue, and refuses to provide any details about the violation. From the customer's perspective, the results are very much as if the cloud provider went out of business. But this behavior cannot be predicted from the normal signal of "a reliable business that will be around for a while".

It may take some time, and a few more stories about sudden, unexplained and uncorrectable account closures, but eventually people (and businesses) will recognize the risk and start taking preventative actions. Actions such as keeping local copies of data, backups of that data (not local and not on the main cloud provider), and a second provider for fail-over.

In other words:

Don't put all of your eggs in one cloud basket.

Is "cheap and easy" a good thing?

In the IT industry, we are all about developing (and adopting) new techniques. The techniques often start as manual processes, often slow, expensive, and unreliable. We automate these processes, and eventually, the processes become cheap and easy. One would think that this path is a good thing.

But there is a dark spot.

Consider two aspects of software development: backups and version control.

More often than I like, I encounter projects that do not use a version control system. And many times, I encounter shops that have no process for creating backup copies of data.

In the early days of PCs, backups were expensive and consumed time and resources. The history of version control systems is similar. The earliest (primitive) systems were followed by (expensive) commercial solutions (that also consumed time and resources).

But the early objections to backups and version control no longer hold. There are solutions that are freely available, easy to use, easy to administer, and mostly automatic. Disk space and network connections are plentiful.

These solutions do require some effort and some administration. Nothing is completely free, or completely automatic. But the costs are significantly less than they were.

The resistance to version control is, then, only in the mindset of the project manager (or chief programmer, or architect, or whoever is running the show). If a project is not using version control, its because the project manager thinks that not using version control will be faster (or cheaper, or better) than using version control. If a shop is not making backup copies of important data, its because the manager thinks that not making backups is cheaper than making backups.

It is not enough for a solution to be cheap and easy. A solution has to be recognized as cheap and easy, and recognized as the right thing to do. The problem facing "infrastructure" items like backups and version control is that as they become cheap and easy, they also fade into the background. Solutions that "run themselves" require little in the way of attention from managers, who rightfully focus their efforts on running the business.

When solutions become cheap and easy (and reliable), they fall off of managers' radar. I suspect that few magazine articles talk about backup systems. (The ones that do probably discuss compliance with regulations for specific environments.) Today's articles on version control talk about the benefits of the new technologies (distributed version control systems), not the necessity of version control.

So here is the fading pain effect: We start with a need. We develop solutions, and make those tasks easier and more reliable, and we reduce the pain. As the pain is reduced, the visibility of the tasks drops. As the visibility drops, the importance assigned by managers drops. As the importance drops, fewer resources are assigned to the task. Resources are allocated to other, bigger pains. ("The squeaky wheel gets the grease.")

Beyond that, there seems to be a "window of awareness" for technical infrastructure solutions. When we invent techniques (version control, for example), there is a certain level of discussion and awareness of the techniques. As we improve the tools, the discussions become fewer, and at some point they live only in obscure corners of web forums. Shops that have adopted the techniques continue to use them, but shops that did not adopt the techniques have little impetus to adopt them, since they (the solutions) are no longer discussed.

So if you're a shop and you're "muddling through" with a manual solution (or no solution), you eventually stop getting the message that there are automated solutions. At this point, it is likely that you will never adopt the technology.

And this is why I think that "cheap and easy" may not always be a good thing.