Friday, July 23, 2010

Mainframes in your pocket

Some time ago, I worked at a bank. This bank, which was small with only fifteen branch offices and assets of less than a billion dollars) ran their business on an IBM 4331 processor with some disc drives, a few tape drives, and a printer. The processor was a refrigerator-sized box that required special power and a temperature-controlled environment.

From what I can tell, the processing power, memory, and disk capacities of their computer room were smaller than that of today's typical smartphone. (Although perhaps not including the printer.) By today's standards, the bank's computing power is miniscule, less that a college student uses for talking to friends and chatting on Facebook.

Yet even with such an increase in computing power, we are still stingy with it. Or more specifically, large organizations have processes and mindsets that are still stingy with it.

Take for example the process to set of a centralized processing point for some data. And assume that the remote points (either external customers or internal teams) have data not on paper forms but on computers of their own. In other words, you are building a service.

The typical process (for a large organization) is: collect requirements, design the system, specify the transfer formats, build the service (and test it), and have each "user" change their system to use the approved data format.

A recent O'Reilly Radar article noted that some large data projects accept and process data in just about any format. That is, they do not force clients to use a specific format. I find this approach refreshing and efficient.

Long-time project managers and system designers might find that statement odd. After all, we all know that the most efficient way to build a system is to use a standard input format. A standard format reduces the work for the development team and allows for easier testing.

But there are other costs to consider.

Within the organization, such a process pushes the cost to each of the consumer teams. Each and every consumer team must convert their data into the standard format. Thus you have not saved costs but multiplied them.

If your clients are all external customers, then you have transferred the cost of formatting to your customers. Such a process is considered the norm, for some reason. Pushing the task of formatting data onto internal teams (or external customers) is given a free pass. If you were to set hoops for other business tasks (such as demanding invoices be in German with amounts in Renminbi, and all other correspondence in Brazilian Portuguese) they other business users would revolt. (Or worse, quietly take their business elsewhere.) Yet the same customers accept the cost of formatting data.

If you have a single client and it is an internal one, then you have transferred the cost of formatting to that team. With only one user, there is no net cost to the organization -- either the service team incurs the cost or the user team incurs the cost. (I'm assuming that the cost is about the same for both teams.)

But if you have multiple user teams, you have pushed the cost onto all of the user teams. Each team must expend some effort to convert their data into the approved format.

This approach is not minimizing cost to the organization; it is distributing costs to different teams (and probably different budgets).

The counter argument runs thus: forcing the serving team to accept all formats increases their development costs, possibly more than the savings in the consumer teams since the serving team must learn the nuances of the different formats. And this argument is also correct -- when the formats are not commonly accepted formats. When the data is sent in proprietary formats, this argument holds.

The argument fails when the data is in common formats (such as XML, Microsoft Office formats, or Open Office, or YAML, ...). Here the formats are well understood.

At this point, the argument becomes: We cannot use those formats; we don't have the processing power to convert our data into them. We need custom formats for efficiency.

Efficiency?

Really?

When college students walk around with more processing power in their pocket than several IBM 370 mainframes?

I find that hard to believe. We have the processing power. It's time to leverage that power to make business processes more efficient.

No comments: