... or the few

Software development is maturing.

I suppose that is has been since the Eldar days, when programming meant wiring plug-boards. Programming languages and tools have been changing since the first symbolic assemblers.

There have been two types of changes ("improvements"?) for programming: changes that benefit the individual programmer and changes that benefit the team.

Changes that benefit the individual include:

 - FORTRAN
 - Unix, which let a programmer use many tools and even build his own
 - Integrated development environments
 - Interactive debuggers
 - Faster processors and increased memory
 - Unit tests (especially automated unit tests)
 - Diagnostic programs such as 'lint'
 - Managed environments with garbage collection (eliminating the need for 'delete' operations)
 - Early forms of version control systems

Changes that benefit the team include:

 - COBOL, with its separation of concerns and support for discrete modules
 - Early operating systems that scheduled jobs and managed common resources
 - The 'patch' program, which allowed for updates without transmitting the entire source
 - Network connections (especially internet connections)
 - Code reviews
 - Distributed version control systems
 - github

One can argue that any of these technologies help individuals and teams. A better debugger may directly help the programmer, yet it helps the team by making the developer more productive. Some technologies are better for the individual, and others better for the team. But that's not the point.

The point is that we, as an industry, look to improve performance for individuals and teams, not just individuals. We look at teams as more than a collection of individuals. If we considered teams nothing more than a group of people working on the same project, we would focus our efforts on individual performance and nothing else. We would develop better tools for individuals, and not for teams.

Some languages are designed for teams, or at least have aspects designed for teams. The Python and Go languages have strong ideas about code style; Python enforces rules for indentation and Go rules for bracket placement. They are not the first; Visual Basic would auto-space and auto-capitalize your source code. That may have been a function of the editor, but since the editor and interpreter were distributed together one can consider the action part of the language.

Python's indentation rules, Go's bracket rules, and Visual Basic's auto-capitalization are all beneficial to the individual programmer, but they are more beneficial to the team. They enforce style upon the source code. Such enforced style ensures that all members of the team (indeed, all members of those programming communities) use the same style. Programmers can more easily move from one project to another, and from code contributed by one person to another. Some programming teams () using other languages) enforce local styles, but these languages have it built into their DNA.

Enforced style is a move against the "cowboy coder", the rebel programmer who insists on doing things "his way". Programming languages are not a democracies -- team members don't get to vote on the rules of the language that they are using -- but it is a step towards government.

Let's count that as maturation.

No easy choice for BYOD

The advent of "bring your own device" (BYOD) poses a problem for companies and their system administrators and support teams. What hardware/software combination shall we choose as the company standard?

In the "old world" of desktop PCs, the choice was easy: Microsoft. Microsoft was the dominant player with the largest market share and the largest set of technologies. Microsoft was also the familiar choice; companies had a lot of Microsoft technology and companies knew that new offerings from Microsoft would work with the existing tech.

The brave new world of tablet/cloud technologies offers us choices, but there is no easy choice. The dominant vendors are not the familiar ones. Microsoft, the familiar choice, has some offerings in cloud technologies but they are not the dominant player. (That role is held by Amazon.com.) With tablets, the leader is Apple. With cell phones, the leader is Apple (or Google with Android, depending on how you look at the numbers). In any of the new tech of phones, tablets, and cloud, Microsoft is not the leader.

If one cannot choose Microsoft, can one at least choose a new single vendor for all the tech? Yes, with some complications. First, the new tech will coordinate poorly -- if at all -- with the existing (probably Microsoft) equipment. Second, the technology lead is split among vendors. Apple may lead in the cell phone and tablet market, but its iCloud offering is significantly less than Amazon.com's AWS offerings. Amazon.com offers the most popular cloud environment, but their tablets are designed for e-books and they have no cell phones. Google has a strong contender with Android for cell phones and tablets, but its cloud offerings are limited compared to AWS and Microsoft's Azure.

The situation is similar to the game of "rock-paper-scissors": no one vendor wins in all categories. If the game depends on cloud tech, then Amazon.com is the safest bet. If instead the game depends on tablets, you may want to use Apple technology. But who can tell the win conditions for the game?

My position is that the choice of a new standard for BYOD is a false one. Or at least the choice of a single set of technology across the company is the wrong choice.

The idea of a single standard, a single set of hardware (and limited software) is a hold-over from the desktop PC model of hardware. In that model, companies purchased, supplied, and supported the hardware and software. When you purchase and supply the equipment, you have to support it. Support costs money (you have to train people) and more configurations means higher costs. Fewer configurations means lower support costs, which is why Southwest Airline operates only Boeing 737 aircraft.

But with BYOD, the support is shifted from the company to the employee. The employee purchases the equipment. They purchase the software. They maintain the device. Forcing a single configuration (or a limited number) is possible, but it saves you nothing. If anything, it will irritate the employees who will see no reason for the restrictions. (It is similar to insisting that your employees all drive cars made by Ford. Unless you are the Ford Motor Company, you have no moral standing for such an edict.)

I recognize that the different tablets and phones run on different platforms. Apps written for iOS must be re-written to run on Android. There are some tools to assist with multi-platform apps. Some apps are best converted to HTML5 and Javascript with style sheets for the different platforms. But this question of platforms is applicable to only your custom, in-house applications. The "big" apps, the popular, general-business apps will move to all of the platforms, just as Facebook and Twitter run on the major platforms. You care about platforms only for your custom applications.

Transitioning from the "we supply everything" model to the "bring your own device" model requires changes for the company and for the employees. It is more complex than a simple "go get your own device" memo. Employees have to take ownership of their devices. Employers must let go of certain decisions and some control. Don't make it harder than it needs to be.

Centralizing or decentralizing? What's happening now?

One might be confused with the direction of today's PC technologies. Are they becoming more distributed or are they becoming more centralized?

There are two large-scale changes in today's technologies. Depending on which you look at, you will see the trend for distributed control or the trend for centralized control. (Kind of like those optical illusions of stairs that go up, or down, depending on your visual perception.)

The distribution of PC applications are becoming centralized: Apple's business model is centralized. Applications for iPhones, iPods, and iPads are distributed through iTunes and through iTunes only. This is very different from the approach used by Microsoft for PC-DOS and Windows applications, in which anyone could write and distribute an application, and anyone could purchase and install an application (provided they had administrator privileges) without any involvement or supervision from Microsoft. Apple has complete control over iTunes and one can distribute an iPad/iPhone/iPod app only with Apple's permission.

Apple has declared intentions to move the Mac world from the "open distributor" model to an "Apple centric" model with the "App Store". Indeed, applications that use iCloud must be distributed within the App Store. Applications distributed via the "open distributor" model cannot use iCloud services.

Microsoft is considering to closed distribution model with Windows 8 and the Metro environment.

The selection and ownership of PCs are becoming decentralized: The "bring your own device" fad (for now, let us call it a fad) shifts the ownership of PCs from employers to employees. The previous model saw employers specifying, providing, and provisioning PCs for workers. Often a company would have a standard configuration of hardware and software, issued to all employees. A standard configuration reduced support costs, since there was one (or a limited number) of hardware and software combinations.

With the "bring your own device" fad, employees pick the device, employees own the device, and employees provision and maintain the device. One person may pick a Windows laptop PC, another may pick a MacBook, and a third may pick a Linux tablet. This is clearly a decentralization of decisions -- although the employer retains decisions for the development of company-specific applications. An employer may develop custom software for their employees and build it for one or a limited number of platforms. (Such as custom software for insurance adjusters that runs only on iPads. You can be sure that insurance adjusters at that company will select iPads.)

The two shifts are symptoms of larger changes: Software is becoming a commodity, with the important packages running on multiple platforms (or equivalents running on different platforms). Second, power is shifting from PC customers (large user corporations) to PC platform manufacturers (Apple, Google, Microsoft).

Software is a commodity, and the different packages offer no compelling advantages. For word processors, Microsoft Office is just as good as Libre Office. And Libre Office is just as good as Microsoft Office. In the past, Microsoft Office did offer compelling advantages: it ran on Windows, it ran efficiently and reliably, and it used proprietary formats that demanded that new uses have the same software. Those advantages have disappeared, for various reasons.

Software is a commodity, and current products are "good enough". There is little to be gained by adding new features to a word processor, to a spreadsheet, to an e-mail/calendar application. I may sound a little like the "we don't need a patent office because we have invented everything" argument, but bear with me.

The core packages used to run offices (word processors, spreadsheets, e-mail, calendars, presentation, etc.) are good enough, the data is interchangeable (or convertable), and the user interfaces are easy enough to understand.

If we need additional functionality in an office, a company will get it (either by building it or buying it). But they will do so with extra software, not through extensions to the core packages. (The one possible exception might be spreadsheet macros.) The core office software are commodities.

Apple knows this. It spends no effort building its own version of office tools. I suspect that Microsoft understands this too, and is preparing for the day when lots of customers move away from Microsoft Office.

Apple and Microsoft are building new mechanisms to extract value from their customers: walled gardens in which they are the gatekeepers. The application software will be less important and the walls around the garden will be more important. Apple uses its iTunes as a tollbooth, extracting a percentage of every sale. I expect Microsoft to do the same.

What this means for "the rest of us" (individuals, user companies, developers, etc.) remains to be seen.

NoSQL is no big deal -- and that is a big deal

Things move fast in tech, or so they say. I saw this effect in action at the OSCON conference, just last week.

I attended this conference last year, so I can compare the topics of interest for this year against last year.

Last year, NoSQL was the big thing. People were talking about it. Vendors were hawking their NoSQL databases. Developers were talking (incessantly, at times) about their projects that use NoSQL databases. Presenters gave classes on the concepts and proper use of NoSQL databases.

This year, no one was talking about it. (Well, a few people were, but they were a tiny minority.) It wasn't that people had rejected NoSQL databases. In fact, quite the opposite: people had accepted them as normal technology. NoSQL databases are now considered to be "just another tool in our set".

So in the space of twelve months, NoSQL has gone from "the cool new thing" to "no big deal".

And that, I think, is a big deal.

How big is "big"?

A recent topic of interest in IT has been "big data", sometimes spelled with capitals: "Big Data". We have no hard and fast definition of big data, no specific threshold to cross from "data" to "big data". Does one terabyte constitute "big data"? If not, what about one petabyte?

This puzzle is similar to the question of "real time". Some systems must perform actions in "real time", yet we do not have a truly standard definition of them. If I design a dashboard system for an automobile and equip the automobile with sensors that report data every two seconds, then a real-time dashboard system must process all of the incoming data, by definition. Should I replace the sensors with units that report data every 1/2 second and the dashboard cannot keep up with the faster rate, then the system is not "real time".

But this means that the definition of "real time" depends not only on the design of the processing unit, but also the devices to which it communicates. The system may be considered "real time" until we change a component, then it is not.

I think that the same logic holds for "big data" systems. Today, we consider multiple petabytes to be "big data". Yet in in 1990 when PCs had disks of 30 megabytes, a data set of one gigabyte would be considered "big data". And in the 1960s, a data set of one megabyte would be "big data".

I think that, in the end, the best we can say is that "big" is as big as we want to define it, and "real time" is as fast as we want to define it. "Big data" will always be larger than the average organization can comfortably handle, and "real time" will always be fast enough to process the incoming transactions.

Which means that we will always have some systems that handle big data (and some that do not), and some systems that run in real time (and some that do not). Using the terms properly will rely not on the capabilities of the core components alone, but on our knowledge of the core and peripheral components. We must understand the whole system to declare it to be "big data" or "real time".

A little knowledge...

Let's start with knowledge. Knowledge on a project can cover a lot of ground, from basic procedures (how to build the programs) to the corporate history and mission and how it affects the design of programs. Some knowledge can be encoded in written procedures, step-by-step lists of operations that provide a specific result. Other knowledge is subtle, and requires good judgement.

This second type of knowledge is hard to acquire and is something that cannot be easily transferred to another person. It is learned only through experience, through trial-and-error, through guidance and mentoring. Knowledge of the first time, in contrast, can be transferred by simply handling a document to a person.

This division of knowledge is not limited to development projects or even IT. Groups such as the Boy Scouts cope with these two types of knowledge. The scout manual covers the basics, but the "good judgement" skills require experience and leadership from other scouts and scoutmasters.

So here is my "great insight":

With a team that has high turnover, the only knowledge that one can expect is the knowledge of the first type, the easily documented procedures. Members of a high-turnover team do not invest enough time in the project to learn the "judgement stuff". Therefore, members of a high-turnover team cannot be expected to "use good judgement" to make decisions and resolve issues; they can only use the documented procedures.

This has ramifications in two areas. One is the documented procedures, the other is the tasks that can be assigned. Documented procedures must be clear and simple, and based on a base of expected experience. They must be of a limited size, something that can be read in a reasonable time. They cannot be a large tome of all possible conditions and desired outcomes -- people will not be able to absorb all of it or navigate to the proper section.

Tasks assigned to a high-turnover team are limited. One cannot assign tasks that require judgement based on knowledge of the corporate culture and project history -- team members will never have this knowledge. (If the outsourcing provider claims that the team can handle such tasks, then the provider must show that the team will have this long-learned knowledge. But I suspect providers will limit their claims to procedure-driven tasks such as coding and execution of test scripts.)

Knowing these limits on outsourcing guides the client company's strategy. With a high-turnover team, the client company must prepare clear and comprehensive procedures for specific tasks. The client can assign procedure-driven tasks but not judgement-driven tasks.

If the client wants to outsource judgement-driven tasks (project management, UI design, test script composition) then the outsourcing company must provide long-term, low-turnover teams. If they cannot, then the client cannot assign those tasks. (At least, not if they want to be successful.)

This time it will be different... for Microsoft, for Apple, and all of us

This month saw another lackluster performance in PC sales. Some folks have indicated that this plateau of sales marks the start of the "Post-PC Era". I think that they are right, except that we will call it the "Tablet/cloud Era". We didn't call the PC Era the "Post Mainframe Era".

This new market is different for Microsoft. They enter as a challenger, not as the leader. They have entered other markets as a challenger (the XBOX is the most obvious example), but mostly they have lived in a market in which they were the top banana. Microsoft struck a fortuitous deal with IBM to supply PC-DOS for the IBM PC, won the OS/2-Windows battle, and had advantages in the development of applications for Windows. (Some have charged that Microsoft had unfair advantages, in that they had detailed technical knowledge of the inner workings of Windows and used that knowledge to build superior offerings, but that debate belongs in the 1990s.)

In the tablet/cloud market, Microsoft enters as a late-comer, after Apple and Google. Microsoft cannot rely on the automatic support of the business market. Success will depend not only on their ability to build hardware and operating systems, but to grow an ecosystem of apps and app developers.

This new market is different for Apple. They enter as the leader, not as a niche provider. Will they keep their lead? Apple has experience as the "odd guy" with a small fraction of the market. Apple is the only manufacturer from the pre-IBM PC age that is still around. Prior to the IBM PC, Apple had a large share but was not a leader. After the IBM PC, Apple kept a small share but remained a side player.

In the tablet/cloud market, Apple is a leader. They provide the most-coveted hardware. Companies copy their designs (and get sued by Apple). Success will require Apple to keep providing new, cool hardware and easy-to-use software.

For Apple and Microsoft, this time it really is different.

A keyboard just for me

We've had the QWERTY keyboard for ages. I learned to type on a real, honest-to-goodness manual typewriter with the QWERTY layout. I learned to program in BASIC on a Teletype ASR-33 (with a QWERTY keyboard), and 8080 assembly on a Heathkit H-89 (with a QWERTY) keyboard. All of these devices had the keyboards built in, part of the device itself.

The IBM PC came with a QWERTY keyboard (at least in the US). Unlike the previous devices, it had a detachable keyboard, and one could replace it with a different layout. (Not that many people did.)

I am sure that some folks still use the Dvorak layout. Some have called for a new standard.

With the invention of the smartphone and the tablet, we now have virtual keyboards. They appear on the display and are composed of programmable bits.

It strikes me that we don't really need a standard keyboard layout. When keyboards were physical things, hard-wired to the main device, a standard layout made sense. (Even when keyboards were mechanical components of manual typewriters, a standard layout made sense.) Physical keyboards could not be changed, and a standard let people easily move from one device to another.

With virtual keyboards we can create individual keyboards and let them follow us from device to device. When keyboards are programmed bits on a screen, it is easy to program those bits for our preferences. We don't need a standard keyboard that everyone agrees to use; better to have our custom keyboard to appear when we use the device.

Those custom keyboards can be any layout. They can be QWERTY. They can be Dvorak. They can be Dextr. They can be any of the above with slight changes. They can be wildly different. They can be programmed with additional keys for characters and glyphs outside of our normal set. (While I use the US layout, I often write the name "Mylène Farmer" and I need the accented 'è'.)

Beyond characters, we can add commonly used words. Android devices often add a key for ".com". We could add custom keys for e-mail addresses. When writing code, we could have special keyboards with keywords of the language ('if', 'while', etc.). Keyboards might interact with development environments. (We see something of this with the smart matching that suggests words as we type.)

I see little need to stay with the QWERTY layout. (Designed, long ago, to prevent people from typing too quickly and jamming the mechanical keys in a manual typewriter.)

Let my keyboard layout be mine!

Punctuated evolution

Biology has the theory of "Punctuated Evolution": periods of stable configurations of species interspersed with short periods of change. I think we can see similar patterns in technology.

Let's start with the IBM PC. It arrived in 1981 and introduced a new age of computing. While preceded by a number of microcomputer systems (the Apple II, the Radio Shack TRS-80, the Commodore PET, and others) the IBM PC gained wide acceptance and set a new standard for computing hardware. The Combination of IBM PC and PC-DOS was the norm from its introduction until 1990 when Microsoft Windows replaced PC-DOS and more importantly, networks arrived.

The PC/Windows/network combination maintained dominance from 1990 until just recently. The PC mutated from a desktop device to a laptop device, and Windows changed from its early incarnation to the Windows-95 and later the Windows Vista "skin". Networks were the thing that really defined this era of computing.

We now have a new transition, from PC/Windows/network to tablet/cloud/wireless. Each transition requires new ideas for processing, storage, and user interfaces, and this transition is no exception. In the PC/DOS era, the user interface was text, the storage was local, and the processing was local. In the PC/Windows/network era, the user interface was graphical, the storage was networked (reliably), and the processing was local.

In the tablet/cloud/wireless era, the user interface is graphical and oriented to touch, the storage is networked (over an unreliable wireless network), and the processing is remote.

The tech for tablet/cloud/wireless is different from the previous age, and requires a different approach to programming and systems design. Processing in the cloud gives us more capacity; communicating over an unreliable network means that our systems must be opportunistic (process when you can) and patient (wait while you cannot).

The PC/DOS era stood for almost twenty years. So did the PC/Windows/network era. If that trend continues, the tablet/cloud/wireless era will run for about the same. Look for tablet/cloud/wireless to run from 2010 to 2030.

Twitter's mistake

Twitter this week decided to stop feeding tweets to LinkedIn. They still allow LinkedIn to feed tweets to Twitter. I think that these moves are mistakes for Twitter. 

When providers of competing services cooperate, the race is to the bottom. That is, to the foundational level, the lowest layer of the system. Microsoft and IBM have been circling each other for years, attempting to cooperate on authentication services. Microsoft is willing to make RACF work with ActiveDirectory, as long as RACF is the authoritative source and ActiveDirectory is merely a client. Microsoft, in turn, is willing to make ActiveDirectory work with RACF, as long as ActiveDirectory is the authoritative source and RACF is the client. Both IBM and Microsoft want to be the base for authentication and are unwilling to yield that position to another.

Similar dances occur in the virtualization world. Microsoft is willing to host Linux in its environments (run by a Windows hypervisor) but is not willing to let Windows run on a foreign hypervisor. (In this specific situation, the dance is asymmetric, as Linux is more than happy to host others or run as a client.)

Twitter, in its move to kick out LinkedIn, has gotten the dance backwards. By turning off the feed to LinkedIn, it has removed itself from the bottom of the hierarchy. (Perhaps "center" is a better description of the desired position among social networks. Let's change our term.)

Twitter and LinkedIn compete, in some sense, in the social network realm. They have different client bases (although with a lot of overlap) and they have different execution models. Twitter users send short (hopefully pithy) blurbs to their followers. LinkedIn users describe themselves and look for business opportunities. Yet both sets of users look for attention, and both Twitter and LinkedIn compete for eyeballs to feed to advertisers.

Okay, that last sentence was a bit more picturesque than I expected. But let's press on.

As I see it, social networks live in an ocean of competitors. They cannot exist on their own -- witness Microsoft's "Live" network that is closed to others. I visit from time to time, but only because I need the Microsoft LiveID. It seems a lonely place.

Social networks like Facebook, Twitter, and LinkedIn tolerate each other. From LinkedIn, I can post messages to my LinkedIn and my Twitter networks. With the Seesmic app, I can post messages to Facebook and Twitter. Sometimes I post a message on only one network; it depends on the tone and content of the message.

The best place for a social network is the center of a person's attention. (This is a game of attention, after all.) Twitter's move pushes me out of Twitter and encourages me to spend more time in LinkedIn, sending messages to LinkedIn and occasionally Twitter. The odds of me sending a Twitter-only message are less than before.

And that's why I think it was a mistake for Twitter.

Our technology shapes our systems

In the old days, computer programs were fairly linear things. They processed data in a linear fashion, and the source code often appeared in a linear fashion.

Early business applications of computers were for accounting applications: general ledger, accounts payable, payroll, inventory... etc. These systems were often designed with a master file and one or more transaction files. The master file held information about customers, accounts, and inventory, and the transaction files held information about, well, transactions, discrete events that changed something in the master file. (For example, a bank's checking accounts would have balances in the master file, and records in the transaction file would adjust those balances. Deposits would increase a balance, checks or fees would decrease a balance.)

The files were not stored on modern devices such as USB drives or even floppy disks. In the early days, the master file was on magnetic tape, and the transactions were on punch cards.

The thing about magnetic tape is that you must run through it from beginning to end. (Much like a tour through an Ikea store.) You cannot jump around from one position to another; you must start with the first record, then process the second record, and in sequence process every record until the last.

The same holds for punch cards. Paper punch cards were placed in a hopper and read and processed one at a time.

You might wonder how you can handle processing of accounts with such restrictions in place. One pass through the master file? And only one pass through the transactions? How can we match transactions to master records if we cannot move to the proper record?

The trick was to align the input files, keeping the master file sorted and sorting the transactions before starting the update process. With a bit of thought, you can imagine a system that reads a master record and a transaction record, compares the account numbers on each (both records need a key for matching) and if they match then updates the master record and moves on to the next transaction. If they don't match then the system stores the master record (on another tape, the output tape) and runs the comparison again. The algorithm does work (although I have simplified it somewhat) and this was a common model for program design.

The rise of direct-access storage devices and complex data structures has changed programming. As processors became less expensive and more powerful, as programming languages became more expressive and allowed complex data structures such as lists and trees, as memory became available to hold complex data structures in their entirety, our model for programming became more complex. No longer were programs limited to the simple cycle of "read-master, read-transaction, compare, update, write-master, repeat".

Programming in that model (perhaps we could call it the "Transaction Pattern") was easy and low-risk because clever people figured out the algorithm and other people could copy it.

This notion of a common system model is not unique to 1960s-style programming. Microsoft Windows programs at the API level follow a specific pattern of messages sent by the Windows core "message pump". Android programs use a similar technique.

Tablet/cloud systems will probably develop one (or perhaps a handful) of common patterns, repeated (perhaps with some variations) for the majority of applications. The trick will be to identify the patterns that let us leverage the platform with minimal thought and risk. Keep your eyes open for common templates for systems. When you find one that works, when you find one that lets lots of people leverage the cleverness of a few individuals, stick with it.

I'm guessing that the system model will not be a purely linear one, as we had in the 1960s. But it may have linear aspects, with message queues serializing transactions and updates.