CrowdStrike, Windows blue screens, and the future

A small problem with CrowdStrike, a Windows security application, has caused a wide-spread problem with thousands, perhaps millions, of PCs running Windows.

Quite a few folks have provided details about the problem, and how it happened.

Instead, I have some ideas about what will happen: what will happen at Microsoft, and what will happen at all of the companies that use CrowdStrike.

Microsoft long ago divided Windows into two spaces: one space for user programs and another space for system processes. The system space includes device drivers.

Applications in the user space can do some things, but not everything. They cannot, for example, interact directly with devices, nor can they access memory outside of their assigned range of addresses. If they do attempt to perform a restricted function, Windows stops the program -- before it causes harm to Windows or another application.

User-space applications cannot cause a blue screen of death.

If an error in CrowdStrike caused a blue screen of death (BSOD), then CrowdStrike must run in the system space. This makes sense, as CrowdStrike must access a lot of things to identify attacks, things normal applications do not look at. CrowdStrike runs with elevated privileges.

I'm guessing that Microsoft, as we speak, is thinking up ways to restrict third-party applications that must run with elevated privileges such as CrowdStrike. Microsoft won't force CrowdStrike into the user space, but Microsoft also cannot allow CrowdStrike to live in the system space where it can damage Windows. We'll probably see an intermediate space, one with more privileges than user-space programs but not all the privileges of system-space applications. Or perhaps application spaces with tailored privileges, each specific to the target application.

The more interesting future is for companies that use Microsoft Windows and applications such as CrowdStrike.

These companies are -- I imagine -- rather disappointed with CrowdStrike. So disappointed that they may choose to sue. I expect that management at several companies are already talking with legal counsel.

A dispute with CrowdStrike will be handled as a contract dispute. But I'm guessing the CrowdStrike, like most tech companies, specified arbitration in their contracts, and limited damages to the cost of the software.

Regardless of contract terms, if CrowdStrike loses, they could be in severe financial hardship. But if they prevail, they could also face a difficult future. Some number of clients will move to other providers, which will reduce CrowdStrike's income.

Other companies will start looking seriously at the contracts from suppliers, and start making adjustments. They will want the ability to sue in court, and they will want damages if the software fails. When the maintenance period renews, clients will want a different set of terms, one that imposes risk upon CrowdStrike.

CrowdStrike will have a difficult decision: accept the new terms or face further loss of business.

This won't stop at CrowdStrike. Client companies will review terms of contracts with all of their suppliers. The "CrowdStrike event" will ripple across the industry. Even companies like Adobe will see pushback to their current contract terms.

Supplier companies that agree to changes in contract terms will have to improve their testing and deployment procedures. Expect to see a wave of interest in process management, testing, verification, static code analysis, and code execution coverage. And, of course, consulting companies and tools to help in those efforts.

Client companies may also review the licenses for open source operating systems and applications. They may also attempt to push risk onto the open source projects. This will probably fail; open source projects make their software available at no cost, so users have little leverage. A company can choose to replace Python with C#, for example, but the threat of "we will stop using your software and pay you nothing instead of using your software and paying you nothing" has little weight.

Therefore shift in contracts will occur in the commercial space, at least not at first. It may change in the future, as changes in the commercial space become the norm.

Streaming services

Streaming services have a difficult business model. The cost of producing (or licensing) content is high, and the revenue from subscriptions or advertisements is low. Fortunately, the ratio of subscribers to movies is high, and the ratio of advertisements to movies is also high. Therefore, the streaming services can balance revenue and costs.

Streaming services can increase their revenue by adjusting subscription fees. But the process is not simple. Raising subscription fees does raise income per subscriber, but it may cause some subscribers to cancel their subscription. Here, economics comes into play, with the notion of the "demand curve", which measures (or attempts to measure) the willingness of customers to pay at different price levels.

Streaming services can decrease their costs by removing content. For licensed content (that is, movies and shows that are made by other companies) the streaming service pays a fee. If they don't "carry" those movies or services, then they don't have to pay. Cancelling their license reduces their cost.

For content that the service produces, the costs are more complex. There is the cost of production, which is a "sunk cost" -- the money has been spent, whether the service carries the movie/show or not. There are also ongoing costs, in the form of residual payments, which are paid to the actors, writers, and other contributors while the movie or show is made available. Thus, a service that has produced a movie can reduce its costs (somewhat) by not carrying said movie.

That's the basic economics of streaming services, a very simplified version. Now let's look at streaming services and the value that they provide to viewers.

I divide streaming services into two groups. Some services make their own content, and other services don't. The situation is somewhat more complicated, because the content-making services also license content from others. Netflix, Paramount+, and Roku all run streaming services, all make their own content, and all license other content to show on their service. Tubi, Frndly, and Pluto TV make no content and simply license content from others.

The content-producing services, in my mind, are the top-tier services. Disney+ makes its own content and buys (permanently) other content to add to its library, and is recognized as a top-tier service. Netflix, Paramount+, and Peacock create their own content (and license some) and I consider them top-tier services.

The services that don't produce content, the services that simply license content and then make it available, are the second-tier services. They are second-tier because their content is available for a limited time. They don't own content; they can only rent it. Therefore, content will be available for some amount of time, and then disappear from the service. (Roku, for example, had the original "Bionic Woman" series, but it is not available now.)

For second-tier services, content comes and goes. There is no guarantee that a specific movie or show will be available in the future. Top-tier services, in contrast, have the ability to keep movies and shows available. They don't, and I think that damages their brand.

Services damage their brand when they remove content, in three ways.

First, they reduce the value of their service. If a service reduces the number of movies and shows available, then they have reduced their value to me. This holds in an absolute sense, and also in a relative sense. If Disney+ removes movies, and Paramount+ keeps its movies, then Disney+ drops in value relative to Paramount+.

Second, they break their image of "all things X". When Paramount+ dropped the series "Star Trek: Prodigy", they lost the right to claim to be home to all things Star Trek. (I don't know that Paramount+ has every made this claim. But they cannot make it now.)

Third, the services lose the image of consistency. On a second-tier service, which lives off of licensed (essentially rented) content, I expect movies and shows to come and go. I expect a top-tier service to be predictable. If I see that it has a movie available this month, I expect them to have it next month, and six months from now, and a year from now. I expect the Disney+ service to have all of the movies that Disney has made over the years, now and in the future. I expect the Paramount+ service to have all of the Star Trek movies and TV shows, now and in the future.

By dropping content, the top-tier services become more like the second-tier services. When Netflix, or Max, or Peacock remove content, they become less reliable, less predictable, less... premium.

Which they may want to consider when setting their monthly subscription rates.

The cloud is not invincible

We tend to think that cloud-based systems as more reliable than our current server-based applications. And they can be, if they are designed properly.

Cloud-based systems use designs that distribute work to multiple servers. Instead of a single web server, you have multiple web servers with some form of load balancing. Instead of a single database server, you have multiple database servers with some form of synchronization. At first glance, it may seem quite similar to a sophisticated system hosted on your in-house servers.

But cloud-based systems are different. Cloud systems have several assumptions:

• the system is distributed among multiple servers
• any layer can be served by multiple servers (there are no special, "magical" servers with unique data)
• any one server may go off-line at any moment
• the cloud can "spin up" a new instance of a server quickly
• requests between servers are queued and can be re-routed to other servers

As long as all of these assumptions hold, we have a reliable system.

Yet the cloud is not invincible. Here are a few ways to build a fragile system:

Require a service all the time (that is, fail if something is not available)

It is easy to fail due to a simple missing service. For example, a web app may have a home page with some information and some widgets. Let's say that one of the widgets is a weather display, showing the current temperature and weather conditions for the user. (We can assume that the widget is informed of the user's location, so that it can request the local weather from the general service.)

If the weather web service is down (that is, not responding, or responding with invalid data), what does your web application do? Does it skip over the weather information (and provide sensible HTML)? Or does it lock in a loop, waiting for a valid response? Or worse, does the page builder throw an exception and terminate?

This problem can occur with internal or external services. Any server can go off-line, any service can become unavailable. How does your system survive the loss of services?

Assume that the cloud will provide servers to meet demand

A big advantage of the cloud is scalability: you get more servers when you need them. And this is true, for the most part. While cloud infrastructure does "scale up" and "scale down" to meet your processing load, the processing power is not guaranteed.

For example, you may have a contract that allows for scaling up to a specified limit. (Such limits are put in place to ensure that the monthly bill will remain within some agreed-upon figure.) If your demand exceeds the contractual limits, your systems will be constrained and your customers may see poor performance.

What warnings will you get about nearing your processing limit? What warnings will your system provide when performance starts to degrade?

Assume that the cloud is the only thing that needs to scale

Even if the cloud infrastructure (servers) scales, does the network capacity? For the big cloud providers, the answer is yes. Does yours?

Designing for the cloud is different than designing for in-house web systems. But not that much different; there is a lot of overlap between them. Use your experience from your existing systems. Think about the problems that cloud computing solves. Learn about the assumptions that no longer hold. Those assumptions work both ways; some are not problems, and some are risks.

You can build reliable cloud-based systems. But don't think that it happens "for free".

Did amazon.com invent cloud computing?

I was the recipient of an interesting idea at the local CloudCamp un-conference.

Amazon.com offers their EC2 virtual servers. At first, it was offered with no guarantees: yeah you can run things on EC2 servers, but they could crash at any time. It was cheap, but not reliable. What was a developer to do?

The price for amazon.com's EC2 service was too low to ignore, so developers did what they always do: they built software around the problems. For cheap servers that could crash at any time, that meant building software that was tolerant of servers "disappearing" at any moment.

That's a big part of cloud computing. With "the cloud", a server can drop off-line at any time. Your application must continue. The initial level of reliability of EC2 was low, and forced developers to think about portable processes, processes that could hop from server to server and continue the work. This idea (along with others) made cloud computing possible.

Prior to the idea of portable processes (and crashing servers), applications were built on the model of "the server is reliable". After EC2, software was built with the idea of "the server is not guaranteed". It's quite a change.

So we may have amazon.com to thank for the cloud.

The "break" button is no more

In the good old days, computers had terminals (clunky ASR-33 Teletypes, or compact LA-34s from DEC, or inexpensive ADM-3As, or the geeky VT-52 with a "bell" that sounded like the grinding of a poorly shifted transmission) and the terminals had buttons labelled "break".

The break button was needed for serial communication lines. We've moved beyond them, using high-speed network connections, and the break buttons (and the terminals too) are things of the past.

We've also lost "break" in another meaning: the more common meaning applied to a device or mechanism that fails to function due to a fault or defective component.

Things break less frequently than they did in the past. What called my attention to this phenomenon was the original version of "Casino Royale", the James Bond movie from the late 1960s. In the movie, there is a car chase (obligatory in a James Bond movie) which involves three cars. Our hero is in one, an evil spy is in a second, and the third car is operated via remote control. (Quite advanced for the 1960s!)

The evil spy's car has a built-in two-way radio, and the third car is controlled via radio by the the same evil spy network. The remote control car has a camera, so the remote control operator can see the front view of the car.

Near the end of the car chase, the two-way radio in the spy's car breaks, and the camera in the remote control car breaks. Bad things happen (to the evil spy) because of the failures. 

The dialog has little to say about these breakages -- just enough, in fact, to let us know that they are broken. And much less than one would expect in a movie of today.

My theory is that audiences of the late 1960s were accepting of the notion of broken devices. So much so that the moviemakers could provide minimal dialog to explain the events, with the expectation that people would understand the failures and "keep up" with the story.

(The contemporary Star Trek series also had many failures, mostly the transporter.)

In contrast, today's movies rarely show things breaking. Devices and gizmos may be deliberately damaged (usually by sword-wielding ninjas or blaster-toting space mercenaries) but things don't break on their own. And people expect things to not break on their own, either in the movies or in real life.

Which means that people expect things to work, that is, to perform as expected. (Until perhaps, attacked by ninjas or mercenaries.) Not just hardware, but software. People expect their PCs to work, their smartphones to work, and the web to work.

When something doesn't work (that is, it's broken), people become frustrated and angry.

If you want happy customers, you had best make sure that your software doesn't break.