Hungarian Notation

I've never liked Hungarian Notation.

No, that's not true. I used my own form of Hungarian Notation in certain C programs (for a certain company). Looking back, I'm not too proud of the  programs. They were inexpertly written, and the strange naming conventions did nothing to improve readability.

I'm pretty sure that the programs have been retired.

I have been reading Numerical Methods and FORTRAN Programming, a work from 1964. It describes programming in FORTRAN and lists (obliquely) the rules for variable names and function names in FORTRAN-IV. They are:

 - Variable and function names are at most six characters
 - Variables of type REAL must begin with a letter in the set A..H, O..Z
 - Variables of type INTEGER must begin with a letter in the set I..N
 - Function names, if longer than three characters, must end in 'F'
 - Variable names cannot end in 'F'
 - Function names must begin with letters that follow the variables, for their return types

These are Hungarian Notation rules, or at least very close in thought. The name of the variable (or function) tells you the data type.

They are different from today's Hungarian Notation, which is a convention agreed to by programmers on a team (or for Visual Basic, a whole industry). These rules are baked into the language! In FORTRAN-IV (and I specify the 'IV' part), you don't declare variables, you simply use them, and the name tells the compiler the type.

FORTRAN 77 came out with a number of improvements, the biggest being the logical IF statement and the dropping of mandatory Hungarian Notation. (Oh, you could still use it, but you could override the "I-through-N are integers" rule. And the 'functions must end in F' nonsense disappeared.)

FORTRAN 77 came out thirty years ago. It had the sense to drop Hungarian Notation. When will the rest of us?

The latest in programming lanugages

Here is late-breaking news about programming languages. The newest, spiffiest, got-to-have-it language is out. It's called LOLcode.

The name comes from the internet LOLcats, since the language is based on their dialect of English. To my knowledge, this is the only programming language based on English since COBOL.

Here is a sample program:

HAI
VISIBLE "I can has cheezburger?"

I HAS A VAR ITZ 1
VISIBLE "I do count!"

IM IN YR LOOP
   VISIBLE VAR
   IZ VAR BIGGER THAN 19 O RLY?
      YA RLY
         GTFO
      NO WAI
         UP VAR!!1
   KTHX
KTHX

KTHXBYE

Like COBOL, all keywords and variable names are in uppercase. And like COBOL, it is meant to be 'readable'. Unlike COBOL, it is meant to be readable by today's up-and-coming internet-savvy, text-phone-using teens, the programmers of tomorrow.

This just may be the next big thing for programming.

Remember, you saw it here first!

Heresy 1: Agile is not fast

It may be a heresy, but I will state it:

Agile Development is not faster than waterfall development.

Programmers using Agile techniques are not faster at adding functionality (or requirements, or function points, or any metric you care to use) than developers using waterfall project planning.

As I see it, Agile techniques offer two advantages: consistent levels of quality and the ability to change you mind. With Agile techniques, the 'test before code' practice ensures that you have tests in place and that all new features work, without breaking existing features. The close contact with users and small steps in development (avoiding the large, "think of everything at once and in the beginning" plans of waterfall) lets you decide on the features to add late in development.

Waterfall makes you commit to functionality up front. Typically, the development organization asks the user organization for a list of changes (in order of priority), estimates the effort for each item, estimates available capacity, and then negotiates on the list of items for the release. This list becomes "the commitment" which is then judiciously guarded by the development organization against changes from the user community.

Agile lets you defer "the commitment" and in fact breaks it down into smaller commitments, usually every week or two. The user community can think about new features and postpone the decision to add them. Rather than one big list decided up front, the list starts small and then adds items in small spurts.

But I see nothing in development practices that make developers go faster. (Well, there are benefits from the test-driven practices, but a waterfall-planned project could use test-driven practices for adding features, so there is nothing unique to Agile.) Programmers add code (features, requirements, function points, what-have-you) at the same rate as development.

I believe that Agile Development is perceived as faster than most waterfall projects, from the effects of test-driven development and the ability to change feature sets. I believe that many users are frustrated with the "committed feature set" of waterfall development (probably the day after the commitment) and welcome the time-delayed possibilities of Agile.

But in the end, the number of features added will be about the same.

So it is not actual performance, but perceived performance that makes Agile development seem faster.

What Techies want

Techies are not so hard to understand. Their wants are simple. Christine Comaford, back in the 1980s, listed four things that techies want:

1) Lavish praise
2) Fun projects to work on and fun tools to work with
3) Free beer and pizza
4) Some money

I liked this list so much I stole it. I repeat it when I can (always giving Ms. Comaford-Lynch credit).

I also modified this list. My modified version is:

1) Lavish, genuine praise
2) Fun projects to work on and fun tools to work with
3) Free beer and pizza, although perhaps not for dinner every night
4) Some money

People (non-techie people, that is) are surprised at the position of money. And at the qualifier "some". Techies don't want lots and lots of money. In general, they want enough to live in reasonable quarters, pay for things like food and the phone bill, and have enough left over for books, music, games, and a few other vices.

It's not that hard.

Lock-in is the new lock-out

What is 'lock-in'? Lock-in is the use of proprietary formats as a way of retaining customers. Microsoft does this with their MS-Word and MS-Excel file formats. The formats are closed, known only to Microsoft and those companies they choose to include in partnerships. But Microsoft is only one example, and I'm not going to pick on Microsoft.

The problem is not limited to Microsoft, and not with Microsoft. The problem is the idea: Lock customer data in a format (or a "prison cell") where only the software from one vendor can read it (has the "key" to unlock the "cell").

So passe! So twentieth century!

Locking data inside an application relies on trust (or naivete) on the part of the customer. A long time ago, IBM salespeople would meet with businessmen, review their use of IBM mainframes and peripherals, and then decide on the next 'generation' of equipment that the business should use. The IBM sales team decided this, alone, not with the customer company. They would then present their recommendations to the customer and frequently the customer would buy everything recommended. The customer trusted the IBM team to recommend the right equipment. (And probably did not have the expertise to disagree.)

Those days are gone. Today, customers for mainframes are savvy and have a keen eye on expenses. And they can pick from a variety of mainframe vendors. When your customers have options, they will pick those options that favor them.

Customers don't (necessarily) want open standards... but they do want to move information from one place to another. Whether documents, spreadsheets, databases, or .mp3 files, customers want to use their data in multiple places, with multiple applications. And they become frustrated when they cannot.

Locking data into a proprietary format can be done when you have a monopoly or near-monopoly. Microsoft has done well with its Office suite, where data is locked inside difficult, if not opaque, file formats. And while some groups have had modest success reverse-engineering those formats, there is always the spectre of incompatibility. The lure of 'file format compatibility' has kept many customers in the Microsoft camp. But a good portion of them chafe. They will jump ship when a new option appears on the horizon. (Some of them will jump simply to get away from the oppressor. But the reason doesn't matter... they leave.)

Microsoft is big and has a commanding presence in the market, and even they have felt the pressure to open their formats. Smaller companies, or companies with smaller market shares (and those companies without 'market power') would be well advised to stay away from the 'lock-in' strategy. It is better to compete on features, usability, and performance.

Of course, that's harder than designing an opaque file format. But I didn't say it would be easy.

No simple answers

Which is the better deal: a 9-inch pizza for $9 or a 12-inch pizza for $14?

The gut-reaction answer is the 9-inch pizza. But that answer is incorrect.

The correct answer is the 12-inch pizza.

Assuming that the pizzas are round, the 9-inch pizza has an area of (almost) 64 square inches and a cost of $0.14 per square inch. The 12-inch pizza has an area of 113 square inches and a cost of $0.12 per square inch.

Why is it easy to pick the wrong answer? Because the two options were presented in terms of diameter, but the value is measured in area. To get the proper answer, you must convert the presented information into the form used for the calculation of value.

Or, in another view, the information presented was 'features' (a 9-inch pizza) and not 'benefits' (total area of pizza). Features are not (necessarily) benefits. Features are attributes of the product. Benefits are things that provide value. They are not the same. One can be converted to the other, provided you know the formula. For pizza, this is an easy conversion. For other items (like software) the conversion can be harder to define.

You knew I was going to get to software at some point, right?

When selecting software, it is easy to pick the one with the most features, on the assumption that it is better than the others. Or at least it will do everything that the other packages will do.

Or it is easy to pick the one with the lowest cost, on the assumption that the possible selections are roughly the same.

Both methods are easy.

And both methods rely on assumptions. Which means they may provide a good result, or may not, depending on the truth to the assumptions.

If you are selecting software, you must understand how the software will be used and how it provides value. You must know the benefits of the software, not merely the features. Only then can you select the best software for your needs.

Knowing how you will use the software is a hard task. It means understanding how people use the software today. It means predicting future use of the software.

It's much easier to simply pick the package with the lowest price. Or the most features.

But the the best solution may not have the lowest price may, nor the most features. Even when they coincide. As an example, consider air travel. If I had to travel from the East coast to the West coast, and were presented with two options: Continental for $299 and Southwest for $349, with departure and arrive times about the same, and Continental offering a free movie and meal on board, I would pick the Southwest flight. (Southwest offers no in-flight movie and provides at most peanuts.)

"Why?", I hear people ask. The Continental flight has a lower cost *and* more features (and other aspects are equal). The cost is better and the feature list is better. In this case isn't Continental the better choice?

The answer is that Southwest is the better choice, because it provides more value. Remember, features are not directly equivalent to value. The features that Continental provides have no value for me: I've eaten their in-flight meals and I find that a sandwich and piece of fruit from the local shop is better.

I've also seen their arrangements for in-flight movies and find it has negative value. Continental uses little flip-down screens that show the movie in multiple locations on there planes; there is no escape from it. A good movie is tolerable but a bad movie is torture. On average, the movie is a disincentive to use Continental. The extra $50 for the Southwest flight is worth it to *not* have a movie inflicted upon me.

There are no simple answers. It's all about the benefits, and identifying the meaningful benefits is work. The simple answers are often wrong.

Which means that the 9-inch pizza might be the better deal, when you are not that hungry.

Two-cycle development

I have been working with software for a while now (more years than I care to mention) and I have found that I work in two different modes. The first mode is 'creating software' and the second is 'refining software'.

The 'creating' mode is when I add new features to a program. It could be at the very beginning of a development effort, when there are no features in a program, or it could be later, when the software is 'mature' (whatever that means).

The 'refining' mode is different. It is not adding features, but changing the software to perform the same feature set but with better internal design. This might mean replacing a sort routine with a better sort routine, refactoring a class for better organization, or changing a program for better division of labor. The important part in this phase is that no features change -- the program works just as before. The internals are different, the externals are the same.

These two modes of thought are, in my mind, part of the basic human psyche. At some times, our minds want to create. At other times, our minds want to re-arrange that which we have created. I consider both modes creative, although different types of creativity.

I think that two-cycle development is necessary. It's necessary because we don't get the design of the program right the first time. Oh, we get something that works, but that doesn't mean the design is right. Despite all of the planning, all of the analysis, and all of the reviews, we build software poorly. Refactoring allows us to examine our construct, think about the design, and figure out better ways to do the job.

Two-cycle development leverages our 'Monday morning quarterbacking' talents (even for our own work), and is possible with automated tests. The ability to re-arrange, to experiment, to try new things, is much easier with a complete set of tests. You need the tests to verify your experiment. Without automated tests, you must rely on manual tests, and manual tests are hard to perform. Since they are hard to perform, programmers tend to skip them. And when programmers skip the tests, they miss errors. After a few experiments in re-organizing code without tests (and a few introduced defects) programmers learn to avoid refactoring operations.

Without automated tests, development tends to avoid refactoring.

Without the refactoring, we use our initial, poor designs. While they work, the designs fail over time. And they often fail in slow ways. Failures occur not in spectacular collapses, but in slow decay. Any system built without refactoring gradually accumulates poor designs, and these poor designs increase the future work on the program. Over time, changes to the program take more time and effort, and the number of defects per change increases. The program gets tangled and hard to understand. Eventually, development grinds to a halt as each change introduces more defects than improvement.

To prevent this 'slow death by poor design', we must allow (and plan for) refactoring. And to refactor the program, we must do to things: allow developers time for refactoring and provide the tests to verify that the changes work. Without these, the program accumulates 'cruft' and becomes harder to maintain.

We interrupt this program...

The worlds of workers and managers are different. Very different.

An observation: Workers work on things, managers work with people. This should not be a surprise. In an automobile assembly plant, the folks on the assembly line (attaching things, adjusting things) are workers. The managers see that people have the tools they need and plan out the work.

For a worker, the 'tools of the trade' are the tools of the trade: hammers, screwdrivers, welding torches, calipers, text editors, SCUBA tanks, ... whatever. Work consists of using these tools to build, fix, or change something.

For managers, the 'tools of the trade' are status meetings, progress reports, and plans. Work consists of talking with others and coordinating activities.

And here we have a significant difference in views.

Workers think of work as 'doing the task'. Managers think of work as 'talking with people'. The talk may be in the form of a conversation, a phone call, or perhaps an e-mail. The tasks of planning and scheduling are secondary to the tasks of communication. Some could argue that a plan is simply a sophisticated form of communication, since it coordinates the work of others.

It's no wonder that managers like status reports and meetings, especially in times of crisis. (Who hasn't been in a crisis where managers asked for daily updates?) Workers, on the other hand, find them distracting.

There is one aspect of communications that is particularly troublesome: interruptions.

I suspect that managers view interruptions as normal, possibly even expected and efficient. Workers view interruptions as... well, interruptions, and consider them detrimental to productivity.

With current technology, interruptions come via a number of channels:

- Walking up to a person and asking "may I ask a quick question?".

- The phone.

- E-mail. Especially when the e-mail system produces noises and graphic alerts that there is a new message.

- Instant messaging.

An interruption requires a 'context switch', a change in thought. When deeply involved in a problem, a worker will give it a lot of thought. Changing the thought to a different topic requires time and effort.

Jerry Weinberg speculates that a context switch reduces efficiency by ten percent. [Quality Software Management, volume 1] This leads to some interesting conclusions: a person working on two tasks (and switching between them) will be able to provide forty-five percent of his time to one and forty-five percent of his time to the other, with the remaining ten percent being 'wasted' in switching from one to the other. The waste increases with the number of tasks.

Project planning software assumes a cost of zero for these switches.

I don't know that I agree with Weinberg's proposed value of ten percent, but I know that I don't buy MS-Project's value of zero. Switching does take time, and the amount of time depends on the complexity of the tasks.

Beyond the cost of switching: Some tasks take time, significant amounts of time. Programmers, working on a problem, will get into 'the zone' where they can focus on the problem (and a solution). They need to stay in that mental state for peak efficiency. 'Significant' will vary with projects, technologies, and tasks, but let us say that a significant amount of time is a period of three hours or more.

If you want a person to work for a significant amount of time, then you cannot interrupt him.

On the other hand, if you (as a manager) expect a person to be interruptible (either by yourself, his co-workers, or others), then you cannot guarantee a period of un-interruptible time, and he will be unable to achieve optimum efficiency.

You cannot have it both ways. Either a person is interruptible (and you pay the penalty for interruptions) or he is not (and you cannot get that 'quick question' answered -- at least not right away).

As I see it, both extremes are bad. Constant interruptions will not allow the worker to finish, and no interruptions will not allow critical questions to be answered. The solution must be a balance, allowing some interruptions but blocking others.

So the question is: What can you do to limit (to a reasonable degree) interruptions?

Canaries in the .NET coal mine

Way back in 2004 (ancient times for the .NET world), Kathleen Dollard wrote an editorial entitled "Save the Hobbyist Programmer", in which she described hobbyist programmers as canaries in coal mines, and further observed that the canaries were dying (figuratively). Hobbyist programmers were not able to keep up with all of the technologies in the .NET universe, and therefore had to move to other arenas.

The editorial provoked a number of responses, some more reasoned than others. Many of them are available on the web (search for "Save the Hobbyist Programmer") although the original editorial is locked inside the Fawcette web site (registration required).

In May of 2007, Kathleen Dollard wrote another editorial, "Pace of Change Leaves No One Competent". In this essay she writes: "No one has time ... to maintain competence in .NET development. ... [T]oday no one can be competent in writing ... modern, high quality applications in any specific genre..."

Both of these editorials appeared in "Visual Studio Magazine", a cheerleader for Microsoft technologies. (The magazine calls itself "Visual Studio Magazine", but most articles are about .NET technologies and not the Visual Studio IDE.)

If the cheerleaders are having discouraging thoughts about .NET technologies, what are the laity thinking?

(To be fair, the second editorial is a retrospective and contains the line "I am more optimistic today". But this line appears a full three-quarters into the article, after a lot of observations on the state of current affairs in the .NET world. And Dollard's suggestion seems to be 'specialize as part of a team', which leaves the hobbyists -- loners, usually -- out in the cold.)

I think the laity are overwhelmed and scared. And they should be: the .NET world is big. Really big. Way too big to learn, and frequently changing. Individuals had a fighting chance of learning C++ and MFC, and certainly could handle Visual Basic in its various incarnations, but .NET is an order of magnitude more complex. (And even Visual Basic required running on a treadmill, with significant changes from one version to the next.)

Microsoft has made attempts to bring hobbyists back to the fold, with the 'Express' versions of products. And while it has worked to an extent, I think Microsoft doesn't get it. Yes, the cost of purchasing products is a barrier for hobbyists, but so is the learning time. Making free editions (crippled or feature-limited) of products does not help the hobbyist when the API contains tens of thousands of entities and changes annually. I don't have time to spend learning an API that I will have to throw away in a year or two. I need to get something working, now. Or at least by the week-end.

Which means that hobbyists must make a choice about .NET: Attempt to learn enough to write small applications, piggy-back hobby efforts onto a day job, or abandon .NET for something else.

Learning enough for small applications is possible, but requires time (and keeping up with changes). A hobby is something I do for enjoyment, in my spare time. It should not be a second job.

Piggy-backing onto my day job may work, provided that my day job deals with enough .NET tech and I have resources to consult. But then my hobby starts looking like my job. Ugh!

Abandoning .NET is not a simple step. For starters, where does one go? Back to MFC? (It had its warts and support will be diminishing in the future.) Change to Java? (That has many of the problems of .NET, although there do seem to be more consistencies.) Switch to Perl, or Python, or Ruby? (They are somewhat more stable, but still require learning.) I don't see a good option.

Maybe I yearn for the "good old days" of BASIC. (Not Visual Basic, but Dartmouth BASIC.) I suppose I could have entitled this rant "The Future isn't the Same as the Past".

But it does scare me that the cheerleaders are complaining.

Promises, promises

What is it about the waterfall model of software development? Why do we use this model?

A little background: The waterfall model has been used in software development for the past forty years, almost as long as we have had software development itself. It consists of breaking a development effort into multiple steps, usually 'analyze', 'design', 'code', 'test', and 'deploy'. (Some shops will have different names for these steps, and some shops will add or remove steps.)

But the waterfall model is accepted as deficient. The seminal paper on the subject, written by Winston Royce, lists this method as possible but he advises against it. (It didn't work in the early days of software development and it works poorly today. The Standish group has a number of studies that show waterfall-run projects are consistently over-budget, late, and result in low quality software.)

Yet we use the waterfall model. And continue to use it, despite the development of other techniques. The current contender is called 'agile' and it works very differently from waterfall.

More background: Agile development discards the notion of 'big design up front' and distinct steps for design, coding, and testing. Instead, it relies on small iterations of deciding on key features, developing automated tests for those features, and then coding the features. The users (or stakeholders) get to decide which features go in first. The development team works on the features until they are complete and then you begin another iteration. You repeat iterations until you have all features you want, and then you ship the product.

The curious reader might be asking "But when do you test? I see no test phase!" Agile development does not require, in fact avoids, a big test effort at the end of the development effort. It puts testing in each iteration, and requires automated testing on each iteration. The 'automated' adjective for testing is important. It allows everyone on the team to run tests, not just a set of test/QA folks. Automated testing also allows *all* tests to be run, so you know that everything you have done to date works. Iteration 2 runs all of iteration 2 tests and all of iteration 1 tests. If a change for iteration 2 breaks something coded in iteration 1, you know about it immediately, and you fix it before proceeding with iteration 3. This discipline is required for the promise that agile development makes.

More on this promise later. For now, I will ramble on about the reasons we keep using waterfall methods.

Some believe that project managers use waterfall because they learned it when they were in school and they stick with it. (Equivalent to 'you cannot teach an old dog new tricks.) Some believe that project managers are capable of learning a new method but see no benefit and are unwilling to risk their career on something untried. Some believe that in large organizations, a sort of 'lock-in' occurs, where all managers use waterfall, expect all other managers to use waterfall, and build there plans around waterfall. When a 'maverick' manager starts using a different process, his plans don't fit in well with the rest of the team, and the rest of the team persuades him to convert his plan (and therefore his process) back to the norm. In this case, the organization has learned waterfall and is unable to learn a new trick.

I believe the reason managers (and organizations) stay with waterfall project plans is because of the promise that a waterfall schedule makes: It promises to deliver X functionality on date Y. (Where X is some random set of agreed-upon features and Y is some agreed-upon date.) This is a very seductive promise; it provides certainty and can be used to 'hold people accountable'.

Agile development methods make a very different promise. They do not promise specific features on a specific date. They promise that the product will always be shippable and that the most important features will be coded first. This promise does not provide the certainty that a set of features will be available on a specific date, which is a very different promise.

Waterfall and Agile address the 'Iron Triangle' (speed, cost, quality -- pick any two) very differently. Waterfall defines speed and cost (and attempts to control quality); agile defines cost and quality (and measures speed but makes no attempt to control it).

Two very different approaches to project planning.