Agile is not for everyone

Agile development is a hot topic for many project managers. It is the latest management fad, following total cost of ownership (TCO) and total quality management (TQM). (Remember those?)

So here is a little bit of heresy: Agile development methods are not for every project.

To use agile methods effectively, one must understand what they offer -- and what they don't.

Agile methods make one guarantee: that your system will always work, for the functionality that has been built. Agile methods (stakeholder involvement, short development cycles, automated testing) ensures that the features you build will work as you expect. Even as you add new features, your automated tests ensure that old features still work. Thus, you can always send the most recent version of your system to your customers.

But agile methods don't make very promise. Specifically, they don't promise that all of your desired features will be available (and working) on a specific date. You may have a list of one hundred features; agile lets you implement those features in a desired order but does not guarantee that they will all be completed by an arbitrary date. The guarantee is only that the features implemented by that date will work. (And you cannot demand that all features be implemented by that date -- that's not agile development.)

Agile does let you make projections about progress, once you have experience with a team, the technology, and a set of features for a system. But these projections must be based on experience, not on "gut feel". Also, the projections are just that: projections. They are estimates and not guarantees.

Certain businesses want to commit to specific features on specific dates, perhaps to deliver a system to a customer. If that is your business, that you should look carefully at agile methods and understand what they can provide. It may be that the older "waterfall" methods, which do promise a specific set of features on a specific date, are a better match.

Google, Oracle, and Java

Apple has a cozy walled garden for its technology: Apple devices running Apple operating systems and Apple-approved apps written in Apple-controlled languages (Objective-C and now Swift).

Microsoft is building a walled garden for its technology. Commodity devices with standards set by Microsoft, running Microsoft operating systems and apps written in Microsoft-controlled languages (C#, F#, and possibly VB.NET). Microsoft does not have the same level of control over applications as Apple; desktop PCs allow anyone with administrator privileges to install any app from any source.

Google has a walled garden for its technology (Android), but its control is less than that of Apple or Microsoft. Android runs on commodity hardware, with standards set by Google. Almost anyone can install apps on their Google phone or tablet. And interestingly, the Android platform apps run in Java, a language controlled by... Oracle.

This last aspect must be worrying to Google. Oracle and Google have a less than chummy relationship, with lawsuits about the Java API. Basing a walled garden on someone else's technology is risky.

What to do? If I were Google, I would consider changing the language for the Android platform. That's not a small task, but the benefits may outweigh the costs. Certainly their current apps would have to be re-written for the New language. A run-time engine would have to be included in Android. The biggest task would be convincing the third-party developers to change their development process and their existing apps. (Some apps may never be converted.)

Which language to pick? That's an easy call. It should be a language that Google controls: Dart or Go. Dart is designed as a replacement for JavaScript, yet could be used for general applications. Go is, in my opinion, the better choice. It *is* designed for general applications, and includes support for concurrency.

A third candidate is Python. Google supports Python in their App Engine cloud platform, so they have some familiarity with it. No one company controls it (Java was controlled by Sun prior to Oracle) so it is unlikely to be purchased.

Java was a good choice for launching the Android platform. I think the languages Go and Python are better choices for Android now.

Let's see what Google thinks.

Technology fragmentation means smaller, simpler systems

In the past, IT shops standardized their technologies, often around the vendor deemed the industry leader. In the 1960s and 1970s, that leader was IBM. They offered products for all of your computing needs, from computers to terminals to printers and even punch cards.

In the 1980s and 1990s, it was Microsoft. They offered products for all of your computing needs, from operating systems to compilers to office suites to user management. (Microsoft offered little in hardware, but then hardware was considered a commodity and available from multiple sources.)

Today, things are not so simple. No one vendor that provides products and services for "all of your computing needs". The major vendors are Microsoft, Apple, Amazon.com, Google, and a few others.

Microsoft has a line of offerings, but it is weak in the mobile area. Sales of Microsoft tablets, Microsoft phones, and Windows Mobile are anemic. Anyone who wants to offer services in the mobile market must deal with either Apple or Google (and preferably both, as neither has a clear lead).

Apple has a line of offerings, but is weak in the enterprise area. They offer tools for development of apps to run on their devices but little support for server-side development. Anyone who wants to offer services that use server-side applications must look to Microsoft or Google.

Amazon.com offers cloud services and consumer devices (the Kindle) but is weak on development tools and transaction databases. Google offers cloud services and consumer devices as well, but lacks the enterprise-level administration tools.

Complicating matters is the plethora of open-source tools, many of which are not tied to a specific vendor. The Apache web server, the Perl and Python languages, several NoSQL databases, and development tools are available but not with the blessings (and support) from vendors.

Development teams must now cope with the following:

Browsers: Internet Explorer, Chrome, Firefox, Safari, and possibly Opera
Desktop operating systems: Windows (versions 7, 8, and 10), MacOS X, Linux (Ubuntu, SuSE, and Red Hat)
Platforms: desktop, tablet, phone
Mobile operating systems: iOS, Android, and possibly Blackberry and Windows
Database technologies: SQL and NoSQL
HTTP servers: Apache, NGINX, and IIS
Programming languages: C#, Java, Swift, Python, Ruby, and maybe C++ or C
Cloud platforms: Amazon.com AWS, Microsoft Azure, Google cloud
Cloud paradigms: public cloud, private cloud, or hybrid

I find this an impressive list. You may have some factors of your own to add. (Then the list is even more impressive.)

This fragmentation of technology affects your business. I can think of several areas of concern.

The technology for your systems You have to decide which technologies to use. I suppose you could pick all of them, using one set of technology for one project and another set of technology for another project. That may be useful in the very short term, but may lead to an inconsistent product line. For example, one product may run on Android phones and tablets (only), and another may run on Apple phones and tablets (only).

Talent for that technology Staffing teams is an on-going effort. If your project uses HTML 5, CSS, JavaScript, and Python with a NoSQL database, you will need developers with that set of skills. But developers don't know everything (even though some may claim that they do) and you may find few with that exact set of technology. Are you willing to hire someone without on of your desired skills and let that person learn them?

Mergers and acquisitions Combining systems may be tricky. If you acquire a small firm that uses native Android apps and a C#/.NET server-side system, how do you consolidate that system into your HTML, CSS, JavaScript, Python shop? Or do you maintain two systems with distinct technologies? (Refer to "inconsistent offerings", above.)

There are no simple answers to these questions. Some shops will standardize on a set of technologies, combining offerings from multiple vendors. Some will standardize on a vendor, with the hope of a future industry leader that sets the standard for the market. Many will probably have heated arguments about their selections, and some individuals may leave, staying more loyal to the technology than the employer.

My advice is to keep informed, set standards when necessary, and keep systems small and simple. Position your technology to shift with changes in the industry. (For example, native apps on Apple devices will shift from Objective-C to Swift.) If your systems are large and complicated, redesign them to be smaller and simpler.

Build and maintain systems with the idea that they will change.

They probably will. Sooner than you think.

Goodbye, Dr. Dobbs

Today the Dr. Dobbs website, sequel of the august publication of the same name, announced that it was going out of business.

It is an announcement that causes me some sadness. Dr. Dobbs was the last of the "originals", the publications of the Elder Days before the web, before the internet, before Windows, and even before the IBM PC.

It was a very different time. Computers were slow, low-powered, expensive, and rare. The personal computers at the time used processors that ran at 1 MHz or 2 MHz, had (perhaps) 64KB of RAM, and often stored data on cassette tapes. A typical computer system cost anywhere from $1000 to $4000 (in 1980 dollars!).

Magazines like Dr. Dobbs were the lifeblood of the industry. They provided news, product announcements, reviews, and articles on theory and on practice. In a world without the internet and web sites, magazines were the way to learn about these strange new devices called computers.

Today, computers are fast, powerful, cheap, and common. So common and so cheap that people leave working computers in the trash. So fast and powerful that we no longer care (much) about the processor speed or memory size.

Not only are computers plentiful, but information about computers is plentiful. Various web sites provide news, opinion, and technical information. We don't need a single "go to" site for all of that information; Google can find it for us.

So, goodbye to Dr Dobbs. You served us well. You helped us through a difficult time, and shared information with many. You were one of the factors in the success of those early days, and therefore the success of the PC industry today. You will be missed.

Software doesn't rot - or does it?

Is it possible for software to rot? At first glance, it seems impossible. Software is one of the more intangible constructs of man. It is not exposed to the elements. It does not rust or decompose. Software, in its executable form, is nothing more than digitally-stored bits. Even its source form is digitally-stored bits.

The hardware on which those bits are stored may rot. Magnetic tapes and disks lose their field impressions, and their flexibility, over time. Flash memory used in USB thumb drives lasts a long time, but it too can succumb to physical forces. Even punch cards can burn, become soggy, and get eaten by insects. But software itself, being nothing more than ideas, lasts forever.

Sort of.

Software doesn't rot, rust, decompose, or change. What changes is the items that surround the software, that interact with the software. These change over time. The changes are usually small and gradual. When those item change enough to affect the software, we notice.

What are these items?

User expectations Another intangible changing relative to the intangible software. Users, along with product owners, project managers, support personnel, and others, all have expectations of software. Those expectations are formed not just from the software and its operating environment, but also from other software performing similar (or different) tasks.

The discovery of defects Software is complex, and more software has some number of defects. We attempt to build software free of defects, but the complexity of computer systems (and the business rules behind computer systems) makes that difficult. Often, defects are built in to the system and remain unnoticed for weeks, months, or even years. The discovery of a defect is a bit of "rot".

The business environment New business opportunities, new markets, and new product lines can cause new expectations of software. Changes in law, from new regulations to a different rate for sales tax, can affect software.

Hardware Software tends to outlive hardware. Moving from one computer to a later model can expose defects. Games for the original IBM PC failed (or worked poorly) on the IBM PC with its faster processor. Microsoft Xenix ran on the Intel 80286 but not the 80386 because it used reserved flags in the processor status word. (The 80286 allowed the use of reserved bits; the 80386 enforced the rules.) The install program for Wordstar, having worked for years, would fail on PCs with more than 512K of RAM (this was in the DOS days), a lurking defect exposed by a change in hardware.

The operating system New versions of an operating system can fix defects in the old version. If application software took advantage of that defect (perhaps to improve performance) then the software fails on the later version. Or a new version implies new requirements, such as the requirements for branding your software "Windows-95 ready". (Microsoft imposed several requirements for Windows 95 and many applications had to be adjusted to meet these rules.)

The programming language Microsoft made numerous changes to Visual Basic. Many new versions broke the code from previous versions, causing developers to scramble and devote time to incorporating the changes. The C++ and C# languages have been stable, but even they have had changes.

Paradigm shifts We saw large changes when moving from DOS to Windows. We saw large changes when moving from desktop to web. We're seeing large changes with the move from web to mobile. We want software to operate in the new paradigm, but new paradigms are quite different from old ones and the software must change (often drastically).

The availability of talent Languages and programming technologies rise and fall in popularity. COBOL was once the standard language of business applications; today there are few who know it and fewer who who teach it. C++ is following a similar path, and the rise of NoSQL databases means that SQL will see a decline in available talent. You can stay with the technology, but getting people to work on it will become harder -- and more expensive.

All of these factors surround software (with the exception of lurking defects). The software doesn't change, but these things do, and the software moves "out of alignment" with the needs and expectations of the business. Once out of alignment, we consider the software to be "defective" or "buggy" or "legacy".

Our perception is that software "gets out of alignment" or "rots". It's not quite true -- the software has not changed. The surrounding elements have changed, including our expectations, and we perceive the changes as "software rot".

So, yes, software can rot, in the sense that it does not keep up with our expectations.

Open source .NET is less special and more welcoming

The Microsoft "toolchain" (the CLR, the .NET framework libraries, and the C# compiler) were special. They were Microsoft's property, guarded jealously and subject to Microsoft's whims. They were also the premiere platform and tools for development in Windows and for Windows. If you were serious about application development (for Windows), you used the Microsoft tools.

There were other toolchains. The Java set includes the JVM and the Java compiler. The major scripting languages (Perl, Python, Ruby, and PHP) each have their own runtime engines and class libraries. None were considered special in the way that the Microsoft toolchain was special. (The other toolchains were -- and still are -- considered good, and some people considered them superior to the Microsoft toolchain, but even most non-Microsoft fans would admit that the Microsoft toolchain was of high quality.)

Microsoft's announcement to open the .NET framework and the C# compiler changes that status. Microsoft wants to expand .NET to the Linux and MacOS platforms. They want to expand their community of developers. All reasonable goals; Microsoft clearly sees opportunities beyond the Windows platform.

What interests me is my reaction to the announcement. For me, opening the .NET framework and moving it to other platforms reduces the "specialness" of .NET. The Microsoft toolchain becomes just another toolchain. It is no longer the acknowledged leader for development on Windows.

The demotion of the Microsoft toolchain is accompanied by a promotion of the Java toolchain. Before, the Microsoft toolchain was the "proper" way to develop applications for Windows. Now, it is merely one way. Before, the Java toolchain was the "rebel" way to develop applications for Windows. Now, it is on par with the Microsoft toolchain.

I feel more comfortable developing a Java application to run on Windows. I also feel comfortable developing an application in .NET to run on Windows or Linux. (Yes, I know that the Linux version of .NET is not quite ready. But I'm comfortable with the idea.)

I think other folks will be comfortable with the idea. Comfortable enough to start experimenting with the .NET framework as people have experimented with the Java toolchain. Folks have created new languages to run under the JVM. (Clojure, Scala, and Groovy are popular ones, and there are lots of obscure ones.) I suspect that people avoided experimenting with the Microsoft toolchain because they feared changes or repercussions from Microsoft. Perhaps we will see experiments with the CLR and the .NET framework. (Perhaps new versions of the IronPython and IronRuby projects, too.)

By opening their toolchain, Microsoft has made it more accessible, technically and psychologically. They have reduced the barriers to innovation. I'm looking forward to the results.

The convergence of web and mobile

It's easy to think of web applications and mobile applications as two distinct entities (or two distinct collections of entities). Web applications are hosted on web servers in data centers and they "run" in a browser. Mobile apps are installed on smartphones and tablets and run on those devices. Web applications are large, complex things and mobile apps are small and easy to use. Web applications are general, running on all major web browsers, and mobile apps are built for specific platforms (iOS, Android, etc.).

Yet the two are converging. Mobile apps are becoming more like web applications, and web applications are becoming more like mobile apps.

Let's start with mobile apps. The "standard model" for an app is a small, lightweight user interface that talks to services on back-end servers. In that sense, a mobile app is very much like a web application, which is a web page that sends requests to back-end servers.

The "standard model" for mobile apps is also that the app is installed once on your device and there it sits, ready for you to use. That's not really true, at least not with many of the apps I use.

The app does sit on my phone, but apps are upgraded frequently. The "good old days" of PC applications (or mainframe applications) saw upgrades to applications on an annual frequency, sometimes longer. Today's mobile apps are updates monthly, and sometimes more frequently. It seems that Twitter sends updates every week!

The upgrades are usually automatic and silent, requiring no intervention from the operator. One can tell when an app has been upgraded (sometimes), because when they display a small "what's new" dialog. A few apps are upgraded more often than I actually use them. I know this because I always see the "what's new" dialog when I run the app.

With upgrade frequencies weekly and approaching daily, perhaps it doesn't make sense to install the app on the phone. That is, perhaps it makes more sense to always download the app from the server. In this way, a mobile app is becoming more like a web page, which is always downloaded from the server. (I'm ignoring cached copies.)

Looking at web applications, we can see changes that make them more like mobile apps. The "standard model" for a web application is that it lives in the browser and has access to only the back-end servers, with no abilities to manipulate data or devices on the host computer. Yet this is no longer true: web apps can upload files, send e-mail (or tie in to local e-mail clients), and now manipulate the local camera and attached devices.

Web applications are becoming more like mobile apps. Mobile apps are becoming more like web applications. Perhaps we will meet in a convenient, happy middle ground that lets us get the best of both worlds.

Maintenance may be more than we think

The effort for systems is often split into "development" and "maintenance". Development is considered the premium of the two: creating a new system to address a business need; maintenance is often considered the "dirty work" of minor adjustments and corrections.

I've been thinking about the terms "development" and "maintenance", and I think our ideas need to change.

Let's consider a non-software example. Let's suppose that we build a hotel in a remote tropical location, on beachfront property. It's a simple hotel with decent rooms and a basic restaurant, but nothing special. With our current understanding of "development" and "maintenance", the construction of such a hotel is clearly "development".

Our hotel is isolated, and far away from other hotels and restaurants.

There are clearly maintenance tasks: cleaning the rooms and hallways, stocking the refrigerators (perhaps one considers that task part of "operations"), and repairing the corrosion from the salty air.

A major change, such as adding air conditioning to the guest rooms or a swimming pool in the common area, is considered "development".

But suppose that we don't add air conditioning or a swimming pool. Suppose we keep our simple hotel, but others buy property nearby and build hotels of their own. (Other people recognize the business opportunity we are enjoying.) And suppose further that they build hotels with air conditioning and swimming pools. We lose some business to the newer, fancier hotels in the area.

Now we're in a different situation. Faced with competition, we're forced to take action. Adding a swimming pool or air conditioning to our hotel is not development, at least not in the sense of defining a new service or market. Enhancing our resort to match the competition is less a matter of expansion and more a matter of stemming the loss of business. In this case, adding a swimming pool is "maintenance".

Development gets you into the market, and maintenance keeps you in the market.

Taking this view of development and maintenance for programs, maintenance becomes a task much larger than simply fixing defects and making minor changes. Maintenance is the act of staying competitive. That includes corrections of defects, and it includes major features to match the competition. It includes adjustments to match changes to the operating system, such as the GUI changes for Windows 8.

Which means that maintenance more than our traditional idea. In one sense it is the old idea of keeping something running. The old sense was focussed on the the thing; the new sense looks at the thing (the program, the hotel, etc.) and the environment. We have to keep the thing running in a changing environment.

One can argue about enhancements to a computer system (or a hotel) and the classification into the categories of "development" and "maintenance". There may be tax ramifications for that classification -- especially for "maintenance contracts". I'm ignoring the tax issues here.

Development and maintenance are perhaps not as distinct as we like to think. If we consider the environment, then enhancements to a system may be either development or maintenance, depending on the surrounding systems.