Implications for Software Engineers
I enjoy watching the Tour de France on TV. It’s fascinating to watch so many cyclists working like crazy, bunched into a small pack, yet not (often) getting their pedals tangled. Ahead of the pack are the leaders, a select group of athletes who pass the famous yellow jersies amongst themselves, stage by stage. They are surrounded by camera cars and excited people on motorcycles. It always amazes me. Here are some of the most aerobically fit people alive, working at the height of their powers, and required to breathe motor exhaust while they do it!
You’ve probably guessed by now that stressing a software engineer is like making the guys in the yellow shirts breathe motor exhaust. The conditions are fundamentally incompatible with top performance. For the engineers it may even be that they can’t do useful work at all (I’ll explain the peculiar minority who sometimes can do useful work later on).
It’s not just engineers who are adversely effected by stress. RAND Corporation report Stress and Performance: A Review of the Literature and Its Applicability to the Military describes how the neuroscience effects cognition and decision making in military contexts. At first you may feel that the stress of being under enemy fire doesn’t equate to office life, but that’s the problem with stress - we have a simple physiological response to sabre toothed tigers, which gets triggered by incoming fire as well as office politics. So it does apply. In addition, although military risk analysis is as rich as a complex software engineering project’s, the end result is always concrete in the extreme, while software engineers are usually dealing with abstracts. Thus the cognitive effects of the equivalent stress are likely to be more pronounced. The report summarizes the effects of stress as making people:
- Screen out peripheral stimuli.
- Make decisions based on heuristics.
- Suffer from performance rigidity or narrow thinking.
- Lose their ability to analyze complicated situations and manipulate information.
In the engineers’ world, every unintended consequence or opportunity to find an error in a design proposal, starts off as a “peripheral stimulus”. Something we haven’t thought of yet. And how often have we heard people saying that they don’t care if something obviously won’t work - what matters is following the procedure! Need I point out that engineers who “Lose their ability to analyze complicated situations and manipulate information” really need to go home until they can?
Let’s make some explicit connections to the reality of programming. Computer Science teacher Mark Guzdial recently asked, What makes programming so hard?. One answer he suggested was:
… it’s the puzzle nature of programming. We’ve all seen puzzles like “Here are 9 dots — cross all of them with four lines” or “Here’s a map with one-ways on it — get from point A to point B using only two left turns and three right turns.” We all know how to make lines and make turns. It’s the puzzle of matching exactly those pieces in just the right way to make it all work.
That certainly sounds like the kind of cognitive flexibility the Beversdorf group talks about. It’s a marriage between variants of algorithms and the kitbags offered by our languages. How can we implement algorithm A in language L? If we modify the algorithm can the implementation be clearer, simpler, more robust? If we can choose the language as well as the implementational details, we have an even richer problem (a vaster search space) to be solved in an “all-or-none insight manner”.
Paul Graham recently wrote one of his typically insightful pieces, Holding a Program in One’s Head, where he says:
A good programmer working intensively on his own code can hold it in his mind the way a mathematician holds a problem he’s working on. Mathematicians don’t answer questions by working them out on paper the way schoolchildren are taught to. They do more in their heads: they try to understand a problem space well enough that they can walk around it the way you can walk around the memory of the house you grew up in. At its best programming is the same. You hold the whole program in your head, and you can manipulate it at will.
Well, that certainly takes a lot of working memory. A person with their working memory impaired (as we know humans and monkeys both are impaired by stress) will not be able to do this. They will have no choice but to work in a focussed attention kind of a way, working it out procedurally, always being able to state each step before they perform it, but with no overview. He goes on to say:
Ordinary programmers working in typical office conditions never enter this mode. Or to put it more dramatically, ordinary programmers working in typical office conditions never really understand the problems they’re solving… There is a contradiction in the very phrase “software company.” The two words are pulling in opposite directions. Any good programmer in a large organization is going to be at odds with it, because organizations are designed to prevent what programmers strive for.
Here I’m arguing that Graham is right about the conflict, but wrong about its origin. It isn’t the interchangability of workers that is the issue (or there would be need for rules about having whole teams travelling in one vehicle). The problem is that so much of corporate motivation, structure and custom is based on stress, pressure, anxiety. For a good example of this, consider the famous Halloween Memorandum, a leaked internal Microsoft document discussing Linux, which included the much ridiculed phrase:
In more forward looking organizations, this structure is provided by strong, visionary leadership.
Is the author really so slavishly devoted to his Glorious Leader that he emits spontaneous sycophancy like this very often? Is it sincere, or does he live in terror of Mr. Gates disembowelling him if he does not thusly bow and scrape? Is the world’s most successful businessman so really so small-minded and insecure that he disembowels insufficiently craven employees? Of course not! But in the corporate context, it is the custom for people to behave as if they live in perpetual fear of disembowelment. And you know what - enough play-acting like that and it starts to become true. Fear becomes conditioned and habitual. This, I think, is a better example of the true conflict between corporatism and programming, and it’s about habituated background stress making engineers neurologically unable to do their jobs.
In Graham’s article he also says:
The danger of a distraction depends not on how long it is, but on how much it scrambles your brain. A programmer can leave the office and go and get a sandwich without losing the code in his head. But the wrong kind of interruption can wipe your brain in 30 seconds.
This is a very important point. Most programmers have at some point experienced the horrific evaporation of their state when they get the wrong kind of interruption. It’s often a clerical worker wanting to have a Kafkaesque argument about a time booking code or something. They tell the clerical worker, “No. I’m busy”, but the clerical worker pays no heed.
Later I’ll discuss the pressures on people to generate stress in organizational contexts, but many of us have seen this one plenty of times. No matter what, you’ve been cornered. You will not escape with your state intact. “But… Just… But… Just…”, splutters the clerical worker, and will not stop until they are told, “NO!”. At which point it is too late. You lose. Hours of state acquisition have been lost, and the programmer is good for nothing for the rest of the day.
In neurological terms we can understand this perfectly. The clerical worker, by subjecting the programmer to bureaucratic stress, has forced the programmer into stress modulated focussed attention, and the ability of his or her brain to hold the program that is already there is lost. Hence “evaporation”. In some organizations it’s quite common for people to suffer this repeatedly, day after day, such that a week goes by with no useful progress - and at the end of it the programmer has suffered aversion therapy such that they start flinch away from trying to load the problem up again.
Of course, the clerical worker has no idea what they have done. For reasons I discuss below, they rarely (if ever) find themselves in anything but focussed attention.
Perhaps the one word which best sums up the faculties so easily blown away by stress is juxtaposition. The ability to bear more than one thing in mind, compare and contrast their structures. Recognizing this, we can see that it’s not just the ability to handle code that is affected. Without juxtaposition, cost/benefit analysis becomes very difficult, and indeed we frequently see people spending weeks implementing “optimizations” that cost more than they save by orders of magnitude. Caching database tables in crude user mode structures on paged systems is a favourite. Then as the application grinds along, they say, “Crikey! Imagine how bad it would be without all the optimization!”, while the more experienced programmers who have a clue what happens inside database engines hold their heads and weep.
Likewise, risk analysis becomes difficult. For example, without the ability to imagine ambiguities in API documentation, portability and compiler compatibility issues, administrative delays in licencing and so on, cost of ownership always ends up just meaning purchase price, and projects become bogged down in fighting unforeseen - but foreseeable - problems caused by “tools” and third party components.
In his article The Perils of JavaSchools, Joel Spolsky refers to “programmers without the part of the brain that does pointers or recursion”. In this “ha ha only serious” comment Spolsky displays a profound intuition. He’s obviously observed his teams very carefully. My only disagreement with him is that I know from experience that everyone has the necessary bit of the brain. It’s just that understanding pointers and recursion requires juxtaposition, and in most people that mode is not accessible.
Similarly, everyone should have the ability to spontaneously notice things going on around them, without being specifically told to look for them. In Godel, Escher, Bach Hofstadter gives the example of the MIU game, which involves transforming strings according to rules. He points out that growing rules are easier to apply than shrinking rules, and that in order to understand the properties of the system, it is necessary to step out of the system - to reflect on what we have seen and spontaneously notice things. Again, reflection requires juxtaposition. A person who is in focussed attention will not be in a position to notice things about the system in this way, and will not see opportunities to simplify or have other insights. This begins to explain the phenomenon of bloatware, both in computer programs and human bureaucracy. Too much stress, and we do need a weatherman to know which way the wind blows!
Going deeper, in The Mythical Man-Month Fred Brooks proposes that conceptual integrity is the essence of good design. How can anyone gauge conceptual integrity if they do not have the capacity to juxtapose at the time?
As a final example, in monograph EWD936, Dijkstra observes that:
Analyzing the structure of traditional mathematical arguments one will discover that the equivalence is the most underexploited logical connective, in contrast to the implication that is used all over the place. The underexploitation of the equivalence, i.e. the failure to exploit inherent symmetries, often lengthens an argument by a factor of 2, 4 or more.
Implication can be followed, worked out, using focussed attention. To utilize or appreciate equivalence requires juxtaposition. To illuminate that which lies behind an observation of Dijkstra is a rare and remarkable thing, and I shall return to this subject later!