Skynet 5 has been fully operational for less than a month, yet already one of our finest minds is chained up in a bunker somewhere, and has been replaced by one of those Terminator skin job thingies.

It would be hard to overstate the contribution of Grady Booch to the young discipline of software engineering. From his own practical experience, his ever-questing mind has observed and crystalized deep wisdom into a collection of writings and products which have benefitted us all. These include the object oriented bits of the Unified Modelling Language (UML), the Rational Rose round-trip tool and his masterwork, Object-Oriented Analysis and Design with Applications. In his definitive book The C++ Programming Language, Bjarne Stroustrup says Booch is the only person worth reading on object oriented design.

So I was astonished to see an interview, Software’s Dirty Little Secret on the Scientific American website. The interview seems wrong-headed in so many ways, yet so typical of the usual unhelpful dross that we see (way below the standard of normal Booch), that I’ve been thinking about it for most of the last few days.

The essence of the interview is hardly novel. The Booch 1000 says that unlike other disciplines, software engineering is a mess because there are no rules for its practitioners to follow. If there was a rulebook, software would not be a mess. That this situation exists is a dirty little secret kept by the software engineers. Oh the shame of it all!

Yes, We Have No Bananas

First off, the claim that:

In other disciplines, engineering in particular, there exist treatises on architecture. This is not the current case in software, which has evolved organically over only the past few decades

is absurd. When I Googled software engineering I got over 54 million hits! We certainly aren’t wanting for rules, proceduralisms, methodologies. Every one of them has been touted as the solution, and still it seems some people can’t understand that there is No Silver Bullet.

Perhaps the Booch 1000 is a member of the this time for sure! school of thought.

No-One Knows Everything

We don’t actually do so badly compared to other disciplines. With thousands of years of prior art behind them, your local builder can construct staircases with good repeatability. We can do the same thing with graphical user interfaces. After all, when Apple implemented the iPhone there wasn’t much doubt that the GUI could be made to work! The builders can do roofing and we can do database engines. In fact, considering some of the debacles I’ve witnessed I’d say we’re rather better at doing database engines than builders do roofing. Builders do commodity guttering, and we do on-the-fly data compression the same way. The fatuous belief that other disciplines are better at this kind of thing comes in part from the fact that builders have to spend time repeating jobs they have done many times before, while often we just link a library. If we typed in the code every time, we’d quickly see the true comparison. (There are some firms which exploit the “business world’s” limited understanding of this reality by typing in the same billing system over and over again, but they are one of the perversities of the field.)

Linking a library is like your local builder following the building regulations. But the building regs only go so far, covering small, localized issues. Outside the limited contexts where the rules can describe the situation exhaustively, the builders can easily get into trouble.

The London Millennium Bridge (image from the Wikipedia article) was a high visibility show-off project using a novel design. As soon as it was opened to the public it revealed a design flaw leading to unpleasant horizontal swaying. Significant modifications were required before people could cross in comfort.

We know about resonant effects. Military formations know they must break step when crossing bridges or they might shake them to bits. Everyone has seen the film of the Tacoma Narrows Bridge:

Beyond bridges, advanced work has been done swinging massive weights around to protect tall buildings from resonant effects during earthquakes. Even so, no-one anticipated the high visibility cock-up in London. As software engineers often do in our young discipline, the designers were trying something just a little new. There was no prior art to guide them, no simple procedures, scripts, building regs for them to follow. And in this case, they ran into trouble. As we often do. The proposition that if only we had a book of rules like the builders do, we’d be perfect like them, is thus shown to be doubly bogus.

Everyone Suffers Feature Interaction

But it doesn’t stop there. Even doing simple things that builders have been doing over and over again for thousands of years, and following the building regs at every point, builders can still be caught out by something we often see too - feature interaction.

These photos below show two reciprocal diagonal views across a pleasant, recently built housing estate that I know. There is a central green space, with houses on two sides:

The other two sides have a stone wall which defines the space and walls the green off from other houses, themselves built around their own greens:

As you can see, the developers went to a lot of effort to create a pleasant and livable space where children can play safely. The houses are well constructed to provide a good quality of life for all. The only problem is a strange acoustic effect. For some reason, the square is like a whispering gallery! The conversations of people walking near to the wall can be clearly heard on the other side of the square, as if they were standing right outside the window. So little energy is lost by whatever focussing effect is in play, that the voices can be heard through the double glazing! At first it’s a very un-nerving experience. It sounds like there’s someone standing there and talking in a muffled way such that their words can’t be discerned, but there’s no-one there! Each bit of the system is in full compliance with building regs, the overall design is a best effort, but this extraordinary interaction of the compliant features happened nevertheless.

Just like what happens when we try new things, this kind of thing cannot be anticipated in building works. In fact if I’d seen a photo of the wall and not known what actually happens, I’d have guessed that it would have anechoic properties:

This is actually an area where software engineers are making better progress - at least there is hope that at some point automated support for CSP theory will reduce our exposure to some kinds of feature interaction in ways that other disciplines cannot hope to achieve.

Then There’s the Cock-Ups Too…

I’ve still not finished my escalation of examples to refute the Booch 1000’s double error, claiming that if we had rules like other disciplines, we’d be predictable and repeatable just like them. The most common cases aren’t like the examples I gave above. Commonly we have rules, just like builders have rules, but at an organizational or regulatory level we choose to ignore them, or follow them in an incredibly stupid, counterproductive way!

The picture below is taken from Google Earth’s view of Stevenage, in south-east England:

There are five roads leading in and out of a roundabout (or traffic circle as amazed Americans call them). If you look closely you will see another network of smaller grey roads threaded adjacent to and flowing under the five wider ones. These are foot and cycle paths. They must have looked wonderful in the artificially constrained context of the pretty watercolours shown to the Planning Department of the local Council. Imagine the atmosphere of self-congratulation as those involved performed their administrative rituals and assured themselves of perfect compliance.

The trouble is, none of those foot and cycle paths are illuminated, and they run like culverts below the level of the road. At dusk they become a muggers’ paradise. Only people with a death wish would venture there. Worse, because of all the wonderful sunken paths, there are no footpaths running by the roadsides. Attempting to walk along the roads exposes people to multiple lanes of high speed traffic. So after dark (which happens by 16:00 hours in winter) there is no pedestrian access to huge areas of the town. It’s like living under a curfew, and as a result Stevenage is a cultural and social desert. And yet… we know all about constructing spaces where people can move about, spontaneously interact, and maintain self-sustaining communities. Spaces where businesses thrive, arts are practiced and a sense of ownership ensures that vandalism doesn’t occur.

The problem is not about having rules. The perpetrators of Stevenage have lots of rules, yet they still accomplished an entirely avoidable cock-up, of the kind which we as software engineers should be interested in, because this is the kind of preventable cock-up which we also commit with monotonous regularity. Even if we allow for the greater call for novelty in software engineering and the amount of feature interaction we are vulnerable to because of the Kolmogorov complexity of our requirements, we still see far too many of these preventable messes.

So unlike the same old, same old proceduralists I deny that we are any worse than other disciplines at achieving repeatable and reliable performance. We have nothing to be ashamed of in this respect, although we are as vulnerable as the others to endemic problems within the culture, and the great promise of our work exposes those problems more.

Don’t Forget - Success is Possible

Architects do sometimes achieve some spectacular successes. Norman Foster’s Millau Viaduct (image from Wikipedia article) is a transcendent blend of form and function that renders further comment superfluous:

I. M. Pei’s Louvre Pyramids (images again from Wikipedia) are an equally masterful blend of an aesthetic which perfectly mirrors the light precision of their context while celebrating the full richness of both eras:

While at the same time magnificently performing the task of filling a vast public area, unimagined when the original Louvre was constructed, with light:

If you haven’t done so yet, go and see it. It’s worth it. As you gaze on these wonders, remember that the vehicular traffic bourne by the one, and the mass of wondering humans hosted by the other, must be measured in the megatonnes. These are very hard working systems.

Getting Proactive

Rather than hanging our heads in shame before the other self-deluded cock-up producers in our culture, and humbly accepting whatever fatuous behavioural prescriptions they hand down to us, we should be taking the lead in sorting out our common problem.

Human culture is a complex multi-layered phenomenon which is capable of great things, but is subject to intermittent faults leading to ridiculous failure. It has something wrong with it and needs debugging. I’m not the only person to notice this - the physicist David Bohm said,

… thought is a system. That system not only includes thought and feelings, but it includes the state of the body; it includes the whole of society - as thought is passing back and forth between people in a process by which thought evolved from ancient times.

He discusses his proposed plan to look for the error in his book, Thought as a System

No rules could have led to the fabulous accomplishments I discussed above, nor yet differentiated between them and the monumental cock-up in Stevenage. Fortunately architecture benefits from its history. Students have to learn plenty of materials science and mechanics, but it is still taught as an Arts subject. The aesthetical judgement - the ability to juxtapose the elements of a problem and composition and see beauty or its absence - of the students is developed. This aesthetical sensibility is shared by great programmers (who use words like elegance to describe it). So I argue that progress in this area will most likely be made by looking at the cognitive state of the practitioners, not the shelfware they execute. It’s about how we perceive and understand the world, and we exercise this kind of paradigmatic flexibility every time we coax a messy set of business requirements into a robust and implementable system design.

We are the systems thinkers, the debuggers, the stars of applied cognitive flexibility. It’s high time we stepped out of our specialization, became fully eclectic in our considerations, and made an effort to sort this stuff out in its own terms - not by just regurgitating the failed excuses already found in the culture.

In the Introduction I propose a picture where psychosocial stress - raised by petty nagging about trivialities - acts to shut down our aesthetical sensibility by pushing us into a kind of focussed attention where working memory is reduced and we lose the juxtapositional ability. We know this happens in lab studies of stress and working memory. As a result we become less able to respond to the circumstances before us and more reliant on heuristics - rules. Beyond that, we know that stress has a similar effect on brain chemistry as addictive drugs, and battery farmed animals subjected to stress become addicted to it. So groups of people can become addicted to nagging each other about petty rules, such that they can only think in terms of rules, and their resulting reduced performance increases the stress. Rule obsession is the problem, not the solution, and it leads to a spiral of decline which ultimately destroys nations and firms.

So we keep on hearing the same old rubbish, always presented as the next big idea, and never mind that it never works. To provide an illustration of this, I’m going to quote a posting to The Joel on Software Discussion Group. The posting describes how things get when groups have free rein to indulge their stress addiction. My proposal is that the behaviours described are attractive not because they are pleasant for the people involved, but because they are stressful. So everyone knows how dreadful it is, but no-one is willing to fix it. Remember that you pay for this twice - once when you pay the victims to behave like this every day, and again when your society suffers the opportunity cost of the kind of maladministration that results. The cognitive limitations have become so widespread that this kind of attitude has become paradigmattic, so that’s it’s now hard to even question why it happens:

- The head of IT has been in government for 27+ years. She is the one making purchasing decisions and setting strategic direction. She does not own a computer or have an external email address. She does not buy on-line, and she has the web monitor set so tight you can forget about using the Internet. No IM. No webmail. Then to ensure you cannot possibly reach the outside, no non-government equipment permitted in the building without a specific exception. This will be your organizational leader as the rule - not the exception.

- Change is glacial. Two weeks I did nothing because they could not allocate a machine for me to access the network, until I had a badge. To get the machine, a form needed to be signed by my boss, his boss, her boss, and his boss. My boss walked it around, and was told that was inappropriate, and it still took 4 hours. Then it went to the help desk who took 9 business days to process and deliver the equipment. Deliver in this case meant coming to my desk and providing a user id and password to a machine sitting there. To install software requires you to have an exception form filed. Another round of 4 signatures in my department, one from security, one from the helpdesk and one from a person who no one can explain except they need to sign the exception form - six days. Complain or bother anyone about a request and it will disappear. Petty is an understatement.

- It is true that no one gets fired. But worse, making decisions is career limiting. The solution is to ensure you never do anything of risk. What you want is to be tied into a project you can claim some responsibility with, if it goes well, and disavow any relationship if it fails.

- No _one_ makes any decisions because there is safety in numbers - very large numbers. All decisions are by committee and expect it to take weeks. They spent 12 weeks, with a minimum of six people in nearly 20 meetings discussing a database key. One key. It is an extreme example but that it can happen says it all.

- If you take any type of leadership position and do anything that employees do not like - they submit a grievance. Why? Because while one is pending, they cannot do anything to you, include request work, or deny automatic or scheduled promotions in grade. A 25 year developer? here has had a grievance on her various leaders, continuously for over 12 years. She is proud of it. “They don’t tell me what to do!”

- Write off leaving government. You can always leave right? Wrong. Would you hire someone from an environment that fosters the behavior above? With very few exceptions, spend more than a year or two as a “gov-e” and you can kiss the private sector good bye. Who would want this behavior and if it took you more than a year to figure out you should quit, that says volumes.

We often see people falling into this swamp. I think it happens to politicians, who often seem sincere when they enter office - and those close to them insist that they are. It doesn’t take long though, before their daily exposure to bureaucrats fixing chronic internally generated psychosocial stress gets to them and they become trapped in the same prescriptive, proceduralist, unimaginative state as the zombie they ousted. And so it goes on.

To conclude this roundup of thoughts triggered by the Grady Booch interview, I was struck by his talking about a dirty little secret. He sounds like a self-hating software engineer!

I have sometimes wondered if there isn’t actually a dirty little secret that is kept by members of groups suffering chronic stress addiction. I suggest that people in this state get a hit off stressful situations and behaviours. They are cognitively impaired and the calibre of debate within the group is poor as a result, but perhaps this provides opportunities to raise stress in a particularly unpleasant way. When a person acts in a manner contradictory to their conscience, this itself is stressful. Might it be that such people are tempted to knowingly engage in wrongful, stupid or damaging behaviours because they know they are performing wrongdoing? The story Pensioner, 81, ordered to remove flat cap as he has a quiet pint - because it’s a security risk is perhaps a mild example of such willful and perverse misinterpretation of guidelines. A kind of willful misinterpretation which is becoming all too common at the moment, as stress addiction becomes deeper and more widespread, and more once bright minds sink into the swamp.