Mind the infrastructure

In our mad rush to the future, it is too easy to become distracted by shiny new tech and ignore the stuff that we already have. Shiny and new is interesting, and the current stuff is boring.

E-mail is probably the most boring of our current systems. Like other utilities, it consistently does its job. It's there for us, all of the time.

Except when it isn't.

I've been working with a national staffing/placement/recruiting company. A big company, one with a name you would recognize.

And their e-mail system is broken.

Not completely broken, but definitely not whole. Their e-mail system has been rejecting messages for three days now. Not just messages from me, but from lots of people outside of their company. Some messages can get through, others cannot. In-bound messages are affected; outbound messages work as you would expect. Internal messages work. But not a good situation, especially in their line of business.

This firm has two problems. The first is that their e-mail system is broken. Like many other companies, they use e-mail for their normal work. With these problems, people must resort to alternative communication means: phone calls, voice mail, and fax. Data transfer is just about impossible, as their placement specialists have no skills with FTP or other data sharing technologies. The loss of e-mail has a significant affect on their business.

The second problem is a little more subtle: Their e-mail system has been broken for three days. Their system administrators are not able to resolve the problem.

Are you starting to worry yet? Here is a company that uses some old, boring tech to run their business. It breaks, and they cannot fix it. (Or they cannot fix it quickly.)

I don't believe that this company will go under because of an e-mail outage. But can any company afford to lose business in the current environment?

It's easy to see how this situation can happen. A company builds an infrastructure. The system works well, but requires expenditures to keep it running. Well-meaning budgeters reduce the allocation of resources. Maybe they lay off the technical lead, or assign him to other projects. Everything still works.

Until it doesn't.

Let's bring this closer to home: What infrastructure do you count on? How well is it supported? Have you cut back on the administration and support teams?

Now are you starting to worry?

Keep an eye on the infrastructure. Know what it is, and make sure that you have people who can keep it running.

iPhone apps raise the bar

The process of installing an application program has changed over time. It started simple, then became complex, then became complicated ("complicated" is worse than "complex"), and now it is simple again.

Installing an application DOS was simple: buy the application on floppy disks, create a directory, copy the files from the one or two floppy disks to that directory, possibly create a batch file, and you were done. The batch file usually used a CD command to change to the directory and then ran the program. Advanced users created batch files to CD back to a common directory when done.

Later DOS programs came with their own install batch files, and sometimes install executable programs. Put in the first disk, type "INSTALL", and away it went! When finished, you could run the program (usually with a batch file that the install program made). But you still had to buy the application on floppy disks, still run the install program, and still interpret the UI.

Microsoft made things consistent in Windows, first by standardizing the name of the install program to "SETUP" and later by standardizing the target location to "C:\Program Files". And during the Windows era, applications grew and moved from floppy disks to CD-ROMs, and later to DVD-ROMs. (But the notion was the same: buy the media, insert, run SETUP, and possibly pick some options.)

The folks at Apple have raised the bar. Not only in the user experience for the application, but the experience of buying an application.

First, with the iTunes store, Apple has created a single place to distribute applications. This is one-stop shopping, with no problems in finding a specific web site for a specific application.

Second, Apple has reduced the steps of purchasing and installing an "app". You go to iTunes, pick your application, pay for it, and it is installed across the network. You don't need a CD. You don't need a system administrator. You don't need a standards committee. You don't have to think about directory names, license keys, or other distractions.

Third, Apple has made applications fun. Beyond the obvious fun apps such as "Labyrinth" and "Ocarina", the basic platform for iPhones, iPods, and Macintoshes encourages applications that please the user. (When where you last pleasantly surprised by a Microsoft application?)

Fourth, Apple has made "apps" small and inexpensive. The small apps can be had for one dollar, many for less than a McDonald's Happy Meal. (Big applications are still expensive.)

After using the Apple platform and "getting spoiled" by the ease at which one can install applications (and do other things like work), the Microsoft platform looks clunky. Microsoft products are big, expensive, install-with-questions beasts.

Microsoft's attempts to hide things from the user (such as with Windows Vista) is well-intentioned and the right direction, yet too many details poke through. The Linux distros (especially SuSE and Ubuntu) have done good jobs, but they also can use some improvement.

Apple has raised the bar, once again, for the user experience. The first question is: can Microsoft and Linux keep up? The second question (for developers) is: can you?

The elusive goal of non-procedural

From time to time, people mention the word "non-procedural". Usually, they are pushing some new technology, some new language or framework, some new thing for developers. The argument is: "In a procedural language, the developer must specify *how* to perform a task; in a non-procedural language, the developer simply specifies *what* must be done."

The implication is that non-procedural languages are more effective for the customer (that is, the customer that the pusher of said technology wants to add to their list). By merely specifying what instead of how, programs can be written faster and with greater reliability.

I have no argument with technology that allows us to write programs faster and more reliably. I myself find that I am faster with C++ than C, and faster with C# than C++, and yet faster with Perl than with C#.

It is the term "non-procedural" that bothers me.

I will argue that third-generation languages (such as the venerable FORTRAN and COBOL) are non-procedural.

"What?!?" you exclaim, "Those old languages are clearly procedural. We have to specify the steps for everything!"

Strictly speaking, that is not true. It is only in assembly language that one must specify the steps for everything. Once you move to a third-level language, the work of register allocation and some data conversion are handled by the compiler. You specify lots of things, but not everything.

So in terms register assignments, compared to assembly language, FORTRAN and COBOL are non-procedural. (You specify *what* not *how*.)

SQL, compared to FORTRAN and COBOL, is non-procedural, in that it handles the input-output operations, comparisons, and sorting.

The issue is not a binary "procedural" or "non-procedural" one. It is a gradient.

Different languages provide different levels of abstraction. SQL provides a (relatively) high level, FORTRAN and COBOL (and C# and Java) an intermediate level, and assembly language a low level. The higher the abstraction, the easier the task.

I predict that at some point in the future we will have a new model for programming databases, one at a higher level than SQL. (Let's call it "uber SQL", just for grins.) The proponents will advertise the ease at which tasks can be accomplished. They will also point to the (then-perceived) flaws with SQL: having to specify how tables are related, and the detailed queries that tell the DBMS how to perform the operation. UberSQL will be considered non-procedural, and SQL will be derided as "too procedural, even for a non-procedural language".

And that is my point. It's not about "procedural" or "non-procedural". It's about abstraction. The higher the abstraction, the more effecient the development. (Because you have to think about less.)

So think in terms of level of abstraction. Avoid the "non-procedural trap", a one-time, "we've done it", error on the way to better programming. We're still developing languages, we're still developing levels of abstractions, and we're still learning how to think.

* * * * *

P.S. Particularly alert readers may take exception with my statement that "only in assembly language that one must specify everything" and suggest machine code as the bottom layer. Other readers may vault past machine code and suggest microcode. I accede to both. I used assembly language to provide a lower level of abstraction. I think the argument against the term "non-procedural" holds, wherever we define the bottom. (Or even if we agree to turtles, all the way down.)

Microsoft wants an ecosystem

Microsoft, for the past several years, has been using the word "ecosystem" to describe the environment of its software. This is a nice ploy by the marketing side of Microsoft but not an accurate one.

I can see why Microsoft wants the word "ecosystem": It implies an inclusive, organic, and dynamic system.

Yet the Microsoft environment is not that inclusive, and one can argue that it is not that organic. (I will grant the "dynamic" portion of the definition.) The Microsoft environment is more engineered than it is organic, and carefully tended. Microsoft programs and tools connect to other Microsoft programs and tools, with occassional connections to the outside. The most obvious example is Windows: Microsoft products run on Windows and nothing else, with the exception of Microsoft Office for the Macintosh. But the containment does not stop there. Microsoft web-based products run in Internet Explorer and possibly other browsers, but Microsoft makes little effort to support them. (The recent addition of FireFox support for Sharepoint is an exception, and it still does not have equivalent status as IE.) Microsoft products such as Sharepoint and SourceSafe connect to SQL Server but not other databases. Windows uses authentication through ActiveDirectory but not LDAP, and ActiveDirectory connects back to SQL Server.

What Microsoft has is more of a biodome than an ecosystem. Of course, the term "ecosystem" is better marketing than "biodome", which implies a confined space with limited options. But a confined space it is. The question is, can you live in it?

Grok around the clock

IBM has a product called "System Grokker" which has a lot of interesting ramifications.

The "Grokker" is a tool that analyzes source code, picks it apart, and presents higher level views. You can read more about it on the IBM web site.

This is similar to a project I was working on, for pretty much the same objective: a better understanding of code by viewing abstract models generated from the code. We built a small (and clumsy) analysis tool that parsed source code, identified classes, found references to classes, and built an affinity graph showing the relationships between classes.

We limited our tool to C, C++, C#, and Java. The parsing task was fairly easy, although each language needed its own parser. The C++ parser was the hardest, which says something about the language. (Templates were the worst part of it, and we ignored generics in C# and Java. We could get away with that -- the code base did not use generics in C# and Java.)

The IBM product creates abstract views of the code. An interesting idea, and well worth examining. The ability to view the system "from a height" is important and underrated. Systems are too large and too complex to view in chunks of twenty lines at a time. An abstract view lets one pull back from the detailed code and see a bigger picture.

But simply pulling back was not all we could do. We built a tree of dependencies (or maybe the term is "graph"). I had to put some "breakers" in place, since the graph can be cyclic (class A refers to class B, which refers to class C, which refers to A). But once I had de-cycled the tree I could calculate the "threat" and "vulnerability" of classes, based on their position in the graph. Classes with no dependencies were not vulnerable to problems from other classes; classes at the top of the graph were vulnerable to changes not just within the class but from their immediate supporting classes and the indirect supporting classes.

We inverted the graph and running similar analysis created a list of "threatening" classes. That is, we identified the classes that were most referenced, the ones that supported the most classes.

With the list of vulnerable and threatening classes, we could focus our efforts on redesign. We also used the list to guide or testing effort, putting more effort on the problem classes.

We also generated a list of cyclic dependencies. We used the list as an aid when re-thinking our design.

But all was not wonderful. The big challenge looming on the horizon is the dynamically-typed languages. Perl, Python, and Ruby don't have the syntactic clues for this kind of analysis. It works for statically-typed languages, where the language provides the information that one needs to build the models. Analyzing dynamically-typed languages will require a different method of collecting class references, possibly  through run-time analysis.

I wish the folks at IBM luck with their project. It is a bold step forward, one that is needed for our craft to improve.

Say, do you think that they are hiring?

Selling Linux

Selling operating systems is tricky. How do you sell something that you can't touch, can't hold, generally gets in the way, and isn't the reason for buying the physical object?

Microsoft sells Windows, but doesn't advertise it. That is, Microsoft offers Windows for sale but its advertisements are generally about other products such as SQL Server or Visual Studio.

Apple sells OSX but like Microsoft, advertises other products like iPods.

Companies such as IBM and Red Hat sell Linux, or more specifically services surrounding Linux.

Geeks "sell" Linux to other computer users, in the sense that they make a sales pitch but they are not getting paid for it.

Here's what's interesting: Geeks use different arguments when selling Linux to fellow geeks and non-geeks.

When convincing a non-geek to switch to Linux, a geek will usually make the following arguments: 1) Linux is reliable and not susceptible to malware; 2) Linux is free (as in zero cost).

When convincing a fellow geek to switch to Linux, a geek may make the above two arguments, but will also make a third argument: "and it's not Microsoft!".

This third argument is different from the previous two, in that it is an emotional argument, not a rational one. Advertising is about emotions, as advertisers learn decades ago. Commercials for cars, beers, fast food, slow food, televisions, and just about everything are based on emotion. The ads tell us about our future life with the product and how happy we will be.

Yet the sales pitch for Linux remains technical and rational. How quaint!

If we want to convince people to try Linux (especially home users), we need to sell to emotions, not technicals. Save the technicals for the geeks -- but throw in the "it's not Microsoft" argument.

Job boards on web 2.0

In technical terms, job boards are medieval.

The purpose of a job board is to connect employeers and candidates, in an efficient manner. Matching employees and candidates is tricky, and the boards that I have seen (and used) do very little to match them.

Boards have two collections: employers and job seekers. Employers must register and create a description of the offered job, and employees can then come and search through the list. Some boards are more sophisticated than others; simple boards list the positions, mainstream boards allow users to search on various criteria (location, skill keyword, date posted), and really savvy boards let job seekers set up agents that perform searches and report results.

The search mechanism is what I find disappointing. It is a keyword-based match, and nothing more. Some job boards allow for multiple keywords and you can specify that all keywords must match. Some job boards fail to recognize certain keywords, such as "C#" or "C++". But the searching is still pretty brute-force.

The problem is that searches yield too many results. A search in a metropolitan area can yield hundreds of matches. One doesn't want to specify too many keywords in fear of locking out a good position that omits a keyword from the description. But too many results is not effective -- after about the twentieth match (of which most are not so great) one's brain becomes fuzzy. The quantity of search results is high. The quality of search results is not so high.

The problem happens on both sides, that is for employers and job seekers. Either can be overwhelmed with responses.

Web 2.0 techniques can help.

Instead of brute-force keyword matches, job boards can let users rate the search responses and use the ratings in future searches. This is the technique used by Netflix to recommend movies.

Instead of movies, job seekers can rate offered jobs. At first (that is, right after registration) the job board knows only the technical skills specified by the user. As the job seeker looks at offered jobs, the user can rate the match and the job board can gain additional knowledge of the job seeker's desired job.

The system works for the employer too, although the job board must keep a set of ratings for each job posted, not a single set. This allows an employer to create a new posting and start with no preset ratings -- which you would want if the previous job was for a COBOL programmer and the new job is for a C# programmer.

The system is not exactly the same as the Netflix algorithm which uses "what other people recommend" -- that method doesn't make sense for job boards. And the overall pool of candidates is changing: as people find jobs they take themselves out of the pool, whereas movies are returned in the Netflix model.

A smarter search should allow for faster matches of employer and job seeker. That is a good thing, if the job board is paid for making such connections. If a job board is paid by advertisers and not by employers or job seekers, then the dynamics change. In that situation, the job board has little incentive for people to find jobs. The incentive is to have people visit the site and see advertisements.

Job boards have been succssful by gaining bulk, that is, the number of openings and the number of job seekers. Some have gained business by using clever advertising. But bulk and advertising is not enough in this world -- you need results. Job boards that deliver lots of poor matches will be at risk of a competitor that delivers fewer high-quality matches.

It's time for job boards to move beyond the medieval approach of brute-force high-quantity results and into the age of intelligent results.