The Cult of Accountability

The disappointing performance of the medical insurance exchange web site (the "Obamacare web site") show the dark side of current project management techniques. After an initial specification, a long wait while multiple teams worked on various parts of the system, and a hasty integration, the web site has numerous, significant problems. Now we have calls from a group I call the "Cult of Accountability". They (the cult) want to know "who is responsible" for the failure.

Big projects often work this way: A large project is assigned to a team (for government projects, the "prime contractor") along with specifications of the deliverable. That team breaks the project into smaller components and assigns components to teams, internal or external (the "sub-contractors") along with specifications for those components. When the work is complete, the work moves in the reverse direction, with the bottom layer of teams providing their components to the next higher layer, those teams assembling the components and providing the results to the next higher layer, until the top team assembles components into a finished product.

This cycle of functional decomposition and specification continues for some number of cycles. Notice that each team starts with a specification, divides the work into smaller pieces, and provides specifications to the down-stream teams.

The top-down design and project planning for many projects is a process that defines tasks, assigns resources, and specifies delivery dates up front. It locks in a deliverable of a specified functionality, a particular design, and a desired level of quality, all on an agreed date. It defines the components and assigns responsibility for each component.

The "divide and conquer" strategy works... if the top team knows everything about the desired deliverable and can divide the work into sensible components, and if the down-stream teams know everything about their particular piece. This is the case for work that has already been done, or work that is very similar to previous work. The assembly of automobiles, for example: each car is a "product" and can be assembled by following well-defined tasks. The work can be divided among multiple teams, some external to the company. The specifications for each part, each assembly, each component, are known and understood.

The "divide and conquer" strategy works poorly for projects that are not similar to previous work. Projects in "unexplored territory" contain a large number of "unknowns". Some are "known unknowns" (we know that we need to test the performance of our database with the expected level of transactions) and some are "unknown unknowns" (we didn't realize that our network bandwidth was insufficient until we went to production). "Unknowns" is another word for "surprises".

In project management, surprises are (usually) bad. You want to avoid them. You can investigate issues and resolve questions, if you know about them. (These are the "known unknowns".) But you cannot (by definition) plan for the "unknown unknowns". If you plan for them, they become "known unknowns".

Project planning must include an evaluation of unknowns, and project process must account for them. Projects with few unknowns can be run with "divide and conquer" (or "waterfall") methods. These projects have few latent surprises.

Projects with many unknowns should be managed with agile techniques. These techniques are better at exploring, performing work in small steps and using the experience from one step to guide later steps. They don't provide a specific date for delivery of all features; they provide a constantly working product with features added over time. They avoid the "big bang" at the end of a long development effort. You exchange certainty of feature set for certainty of quality.

The Cult of Accountability will never accept agile methods. They must have agreements, specific and detailed agreements, up front. In a sense, they are planning to fail -- you need the agreements only when something doesn't work and you need a "fall guy". With agile methods, your deliverable always works, so there is no "accountability hunt". There is no need for a fall guy.