How big is "big"?

A recent topic of interest in IT has been "big data", sometimes spelled with capitals: "Big Data". We have no hard and fast definition of big data, no specific threshold to cross from "data" to "big data". Does one terabyte constitute "big data"? If not, what about one petabyte?

This puzzle is similar to the question of "real time". Some systems must perform actions in "real time", yet we do not have a truly standard definition of them. If I design a dashboard system for an automobile and equip the automobile with sensors that report data every two seconds, then a real-time dashboard system must process all of the incoming data, by definition. Should I replace the sensors with units that report data every 1/2 second and the dashboard cannot keep up with the faster rate, then the system is not "real time".

But this means that the definition of "real time" depends not only on the design of the processing unit, but also the devices to which it communicates. The system may be considered "real time" until we change a component, then it is not.

I think that the same logic holds for "big data" systems. Today, we consider multiple petabytes to be "big data". Yet in in 1990 when PCs had disks of 30 megabytes, a data set of one gigabyte would be considered "big data". And in the 1960s, a data set of one megabyte would be "big data".

I think that, in the end, the best we can say is that "big" is as big as we want to define it, and "real time" is as fast as we want to define it. "Big data" will always be larger than the average organization can comfortably handle, and "real time" will always be fast enough to process the incoming transactions.

Which means that we will always have some systems that handle big data (and some that do not), and some systems that run in real time (and some that do not). Using the terms properly will rely not on the capabilities of the core components alone, but on our knowledge of the core and peripheral components. We must understand the whole system to declare it to be "big data" or "real time".

A little knowledge...

Let's start with knowledge. Knowledge on a project can cover a lot of ground, from basic procedures (how to build the programs) to the corporate history and mission and how it affects the design of programs. Some knowledge can be encoded in written procedures, step-by-step lists of operations that provide a specific result. Other knowledge is subtle, and requires good judgement.

This second type of knowledge is hard to acquire and is something that cannot be easily transferred to another person. It is learned only through experience, through trial-and-error, through guidance and mentoring. Knowledge of the first time, in contrast, can be transferred by simply handling a document to a person.

This division of knowledge is not limited to development projects or even IT. Groups such as the Boy Scouts cope with these two types of knowledge. The scout manual covers the basics, but the "good judgement" skills require experience and leadership from other scouts and scoutmasters.

So here is my "great insight":

With a team that has high turnover, the only knowledge that one can expect is the knowledge of the first type, the easily documented procedures. Members of a high-turnover team do not invest enough time in the project to learn the "judgement stuff". Therefore, members of a high-turnover team cannot be expected to "use good judgement" to make decisions and resolve issues; they can only use the documented procedures.

This has ramifications in two areas. One is the documented procedures, the other is the tasks that can be assigned. Documented procedures must be clear and simple, and based on a base of expected experience. They must be of a limited size, something that can be read in a reasonable time. They cannot be a large tome of all possible conditions and desired outcomes -- people will not be able to absorb all of it or navigate to the proper section.

Tasks assigned to a high-turnover team are limited. One cannot assign tasks that require judgement based on knowledge of the corporate culture and project history -- team members will never have this knowledge. (If the outsourcing provider claims that the team can handle such tasks, then the provider must show that the team will have this long-learned knowledge. But I suspect providers will limit their claims to procedure-driven tasks such as coding and execution of test scripts.)

Knowing these limits on outsourcing guides the client company's strategy. With a high-turnover team, the client company must prepare clear and comprehensive procedures for specific tasks. The client can assign procedure-driven tasks but not judgement-driven tasks.

If the client wants to outsource judgement-driven tasks (project management, UI design, test script composition) then the outsourcing company must provide long-term, low-turnover teams. If they cannot, then the client cannot assign those tasks. (At least, not if they want to be successful.)

This time it will be different... for Microsoft, for Apple, and all of us

This month saw another lackluster performance in PC sales. Some folks have indicated that this plateau of sales marks the start of the "Post-PC Era". I think that they are right, except that we will call it the "Tablet/cloud Era". We didn't call the PC Era the "Post Mainframe Era".

This new market is different for Microsoft. They enter as a challenger, not as the leader. They have entered other markets as a challenger (the XBOX is the most obvious example), but mostly they have lived in a market in which they were the top banana. Microsoft struck a fortuitous deal with IBM to supply PC-DOS for the IBM PC, won the OS/2-Windows battle, and had advantages in the development of applications for Windows. (Some have charged that Microsoft had unfair advantages, in that they had detailed technical knowledge of the inner workings of Windows and used that knowledge to build superior offerings, but that debate belongs in the 1990s.)

In the tablet/cloud market, Microsoft enters as a late-comer, after Apple and Google. Microsoft cannot rely on the automatic support of the business market. Success will depend not only on their ability to build hardware and operating systems, but to grow an ecosystem of apps and app developers.

This new market is different for Apple. They enter as the leader, not as a niche provider. Will they keep their lead? Apple has experience as the "odd guy" with a small fraction of the market. Apple is the only manufacturer from the pre-IBM PC age that is still around. Prior to the IBM PC, Apple had a large share but was not a leader. After the IBM PC, Apple kept a small share but remained a side player.

In the tablet/cloud market, Apple is a leader. They provide the most-coveted hardware. Companies copy their designs (and get sued by Apple). Success will require Apple to keep providing new, cool hardware and easy-to-use software.

For Apple and Microsoft, this time it really is different.

A keyboard just for me

We've had the QWERTY keyboard for ages. I learned to type on a real, honest-to-goodness manual typewriter with the QWERTY layout. I learned to program in BASIC on a Teletype ASR-33 (with a QWERTY keyboard), and 8080 assembly on a Heathkit H-89 (with a QWERTY) keyboard. All of these devices had the keyboards built in, part of the device itself.

The IBM PC came with a QWERTY keyboard (at least in the US). Unlike the previous devices, it had a detachable keyboard, and one could replace it with a different layout. (Not that many people did.)

I am sure that some folks still use the Dvorak layout. Some have called for a new standard.

With the invention of the smartphone and the tablet, we now have virtual keyboards. They appear on the display and are composed of programmable bits.

It strikes me that we don't really need a standard keyboard layout. When keyboards were physical things, hard-wired to the main device, a standard layout made sense. (Even when keyboards were mechanical components of manual typewriters, a standard layout made sense.) Physical keyboards could not be changed, and a standard let people easily move from one device to another.

With virtual keyboards we can create individual keyboards and let them follow us from device to device. When keyboards are programmed bits on a screen, it is easy to program those bits for our preferences. We don't need a standard keyboard that everyone agrees to use; better to have our custom keyboard to appear when we use the device.

Those custom keyboards can be any layout. They can be QWERTY. They can be Dvorak. They can be Dextr. They can be any of the above with slight changes. They can be wildly different. They can be programmed with additional keys for characters and glyphs outside of our normal set. (While I use the US layout, I often write the name "Mylène Farmer" and I need the accented 'è'.)

Beyond characters, we can add commonly used words. Android devices often add a key for ".com". We could add custom keys for e-mail addresses. When writing code, we could have special keyboards with keywords of the language ('if', 'while', etc.). Keyboards might interact with development environments. (We see something of this with the smart matching that suggests words as we type.)

I see little need to stay with the QWERTY layout. (Designed, long ago, to prevent people from typing too quickly and jamming the mechanical keys in a manual typewriter.)

Let my keyboard layout be mine!

Punctuated evolution

Biology has the theory of "Punctuated Evolution": periods of stable configurations of species interspersed with short periods of change. I think we can see similar patterns in technology.

Let's start with the IBM PC. It arrived in 1981 and introduced a new age of computing. While preceded by a number of microcomputer systems (the Apple II, the Radio Shack TRS-80, the Commodore PET, and others) the IBM PC gained wide acceptance and set a new standard for computing hardware. The Combination of IBM PC and PC-DOS was the norm from its introduction until 1990 when Microsoft Windows replaced PC-DOS and more importantly, networks arrived.

The PC/Windows/network combination maintained dominance from 1990 until just recently. The PC mutated from a desktop device to a laptop device, and Windows changed from its early incarnation to the Windows-95 and later the Windows Vista "skin". Networks were the thing that really defined this era of computing.

We now have a new transition, from PC/Windows/network to tablet/cloud/wireless. Each transition requires new ideas for processing, storage, and user interfaces, and this transition is no exception. In the PC/DOS era, the user interface was text, the storage was local, and the processing was local. In the PC/Windows/network era, the user interface was graphical, the storage was networked (reliably), and the processing was local.

In the tablet/cloud/wireless era, the user interface is graphical and oriented to touch, the storage is networked (over an unreliable wireless network), and the processing is remote.

The tech for tablet/cloud/wireless is different from the previous age, and requires a different approach to programming and systems design. Processing in the cloud gives us more capacity; communicating over an unreliable network means that our systems must be opportunistic (process when you can) and patient (wait while you cannot).

The PC/DOS era stood for almost twenty years. So did the PC/Windows/network era. If that trend continues, the tablet/cloud/wireless era will run for about the same. Look for tablet/cloud/wireless to run from 2010 to 2030.

Twitter's mistake

Twitter this week decided to stop feeding tweets to LinkedIn. They still allow LinkedIn to feed tweets to Twitter. I think that these moves are mistakes for Twitter. 

When providers of competing services cooperate, the race is to the bottom. That is, to the foundational level, the lowest layer of the system. Microsoft and IBM have been circling each other for years, attempting to cooperate on authentication services. Microsoft is willing to make RACF work with ActiveDirectory, as long as RACF is the authoritative source and ActiveDirectory is merely a client. Microsoft, in turn, is willing to make ActiveDirectory work with RACF, as long as ActiveDirectory is the authoritative source and RACF is the client. Both IBM and Microsoft want to be the base for authentication and are unwilling to yield that position to another.

Similar dances occur in the virtualization world. Microsoft is willing to host Linux in its environments (run by a Windows hypervisor) but is not willing to let Windows run on a foreign hypervisor. (In this specific situation, the dance is asymmetric, as Linux is more than happy to host others or run as a client.)

Twitter, in its move to kick out LinkedIn, has gotten the dance backwards. By turning off the feed to LinkedIn, it has removed itself from the bottom of the hierarchy. (Perhaps "center" is a better description of the desired position among social networks. Let's change our term.)

Twitter and LinkedIn compete, in some sense, in the social network realm. They have different client bases (although with a lot of overlap) and they have different execution models. Twitter users send short (hopefully pithy) blurbs to their followers. LinkedIn users describe themselves and look for business opportunities. Yet both sets of users look for attention, and both Twitter and LinkedIn compete for eyeballs to feed to advertisers.

Okay, that last sentence was a bit more picturesque than I expected. But let's press on.

As I see it, social networks live in an ocean of competitors. They cannot exist on their own -- witness Microsoft's "Live" network that is closed to others. I visit from time to time, but only because I need the Microsoft LiveID. It seems a lonely place.

Social networks like Facebook, Twitter, and LinkedIn tolerate each other. From LinkedIn, I can post messages to my LinkedIn and my Twitter networks. With the Seesmic app, I can post messages to Facebook and Twitter. Sometimes I post a message on only one network; it depends on the tone and content of the message.

The best place for a social network is the center of a person's attention. (This is a game of attention, after all.) Twitter's move pushes me out of Twitter and encourages me to spend more time in LinkedIn, sending messages to LinkedIn and occasionally Twitter. The odds of me sending a Twitter-only message are less than before.

And that's why I think it was a mistake for Twitter.

Our technology shapes our systems

In the old days, computer programs were fairly linear things. They processed data in a linear fashion, and the source code often appeared in a linear fashion.

Early business applications of computers were for accounting applications: general ledger, accounts payable, payroll, inventory... etc. These systems were often designed with a master file and one or more transaction files. The master file held information about customers, accounts, and inventory, and the transaction files held information about, well, transactions, discrete events that changed something in the master file. (For example, a bank's checking accounts would have balances in the master file, and records in the transaction file would adjust those balances. Deposits would increase a balance, checks or fees would decrease a balance.)

The files were not stored on modern devices such as USB drives or even floppy disks. In the early days, the master file was on magnetic tape, and the transactions were on punch cards.

The thing about magnetic tape is that you must run through it from beginning to end. (Much like a tour through an Ikea store.) You cannot jump around from one position to another; you must start with the first record, then process the second record, and in sequence process every record until the last.

The same holds for punch cards. Paper punch cards were placed in a hopper and read and processed one at a time.

You might wonder how you can handle processing of accounts with such restrictions in place. One pass through the master file? And only one pass through the transactions? How can we match transactions to master records if we cannot move to the proper record?

The trick was to align the input files, keeping the master file sorted and sorting the transactions before starting the update process. With a bit of thought, you can imagine a system that reads a master record and a transaction record, compares the account numbers on each (both records need a key for matching) and if they match then updates the master record and moves on to the next transaction. If they don't match then the system stores the master record (on another tape, the output tape) and runs the comparison again. The algorithm does work (although I have simplified it somewhat) and this was a common model for program design.

The rise of direct-access storage devices and complex data structures has changed programming. As processors became less expensive and more powerful, as programming languages became more expressive and allowed complex data structures such as lists and trees, as memory became available to hold complex data structures in their entirety, our model for programming became more complex. No longer were programs limited to the simple cycle of "read-master, read-transaction, compare, update, write-master, repeat".

Programming in that model (perhaps we could call it the "Transaction Pattern") was easy and low-risk because clever people figured out the algorithm and other people could copy it.

This notion of a common system model is not unique to 1960s-style programming. Microsoft Windows programs at the API level follow a specific pattern of messages sent by the Windows core "message pump". Android programs use a similar technique.

Tablet/cloud systems will probably develop one (or perhaps a handful) of common patterns, repeated (perhaps with some variations) for the majority of applications. The trick will be to identify the patterns that let us leverage the platform with minimal thought and risk. Keep your eyes open for common templates for systems. When you find one that works, when you find one that lets lots of people leverage the cleverness of a few individuals, stick with it.

I'm guessing that the system model will not be a purely linear one, as we had in the 1960s. But it may have linear aspects, with message queues serializing transactions and updates.