Written by Jamie Friday, 27 August 2010 11:21
What it comes to, finally, is that the nation has spent a large part of its time and energy looking away from one of the principal facts of its life. This failure to look reality in the face diminishes a nation as it diminishes a person.
Baldwin, ‘Nobody Knows My Name’.
Greenwood Avenue runs in parallel to Endike Lane, where I was brought up. The capillaries that run off them are where all the action was, the place where the kids hung out. But it was Endike and Greenwood that formed the upper and lower boundaries of the box that was the southerly end of North Hull Housing Estate. Connecting Greenwood and Endike is Cranbrook Avenue. This is where I’d have walked, turning a full 90 degrees right at the first corner and a full 90 degrees left at the second. I passed Ada Holme’s Circle, which housed old people in supervised apartments, before reaching a single-story building that looked like it was built during the war, as a temporary measure, to administer the doling out of ration coupons. The children’s section was in the back, but the lady used to let me take books from the adult section if I read them responsibly. I remember her letting me take a book called Amtrak Wars, and I remember sex being described as ‘putting the bomb in the barrel’. To my amazement, that is exactly how sex is described. I tracked down the book, my curiousity getting the better of me. ‘putting the bomb in the barrel’ appears three times. For example: 'As he moved the zip tag towards Lundkwist she laid a finger on the back of his hand and drew a slow, exploratory circle. Their eyes met. "How about putting the bomb in the barrel"'? (Tiley, 1985)
The summers were warm, somehow different to how they are now. There were less cars, more stray dogs, the threat of violence was much realer, but like the hangover I had a few days ago, seems to me now to be not quite as bad as I know it was.
*
This good old day reverberates through my psyche, its deep, distant echo an indication of the space between me and it. It’s as much a part of me as the scar on my left leg or the anarchy badge I wear upside down on my blue jacket. It guides my ideas. It comes to work with me. It changed the course of my life.
I have an older brother, his name is Brendan. He was born in 1975, on the day of the epiphany. He was like all older brothers, sometimes a bastard, sometimes someone to hang out with. But, at all times, he was in charge of the Commodore 64. This is how I know he was out when I returned from the library. I took the book, an introduction to computer programming, and started typing at the command prompt. The program I chose was a game that manipulated the output to the screen - it made two squares flash. If the squares were the same, the user hit the space bar, got some points, and that was it. All that was required was one programmer (and the author of the book). One project manager. One requirement’s analyst. One time-slice of the hardware bottleneck. In other words, all that was required was me, the bedroom, our Brendan to be out and our mother to be at work.
After that day, I didn’t program for another ten years. I picked up where I left off but instead of writing this:
10 print Jamie
20 goto 10
I wrote this:
for(int i=0; i<1000; i++)
{
System.out.println("Jamie");
}
I worked in a state of oblivion, what Maslov called ‘Unconscious Incompetence’. Donald Rumsfeld tried his best to explain this epistemological idea at a press conference in 2002.
There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. These are things we do not know we don’t know. http://www.brainyquote.com/quotes/quotes/d/donaldrums148142.html
Financial engineers, their teams, and their parent organisations often function at the same level of awareness as I did when I was 10; they do not know what they do not know. They have no idea that they are programmers. Have no idea they are producers and consumers of software. They have no idea that the problems internal customers provide will be solved programmatically, and therefore they have no idea how useful requirements engineering or software management could be.
*
In the fifties my grandmother bought a washing machine complete with mangler, the sort you plugged into the wall, not the sort you turned by hand. The machine loaded from the top. Previous technologies were a washboard and a hand driven mangler. With the new technologies, and if you knew what you were doing, you could do the wash with much less effort. The problem of the wash had grown steadily and linearly in terms of complexity and quantity; my mother is the second youngest of eleven children. The family were left with three management strategies.
My grandmother hated the new system just as I hated Java. She had gotten her arm caught in the mangler and didn’t know how to turn it off, I had gotten confused and afraid of ‘public static void main(String args[]) - I just wanted 20 goto 10. Of course, the technologies were not really the problem. The problem was our abilities, knowledge and expertise. Therefore, what we really hated was ourselves. This is the lesson of my first months at university. This is the lesson of the mangler. As the problems we try to solve evolve, so too must our understanding of them and the tools and technologies we can employ to assist us. This awareness - fourteen years after the event - hangs around me like a curse, my world view changed forever, the childlike naivety of the lone programmer slain and splashed to the four winds of heaven. (Wells, 1889)
*
Sophisticated solutions come from sophisticated, and, importantly, well managed teams. Granny Senior could have reduced the complexity of her work by refusing to wash coloured clothes or by getting rid of some of her kids. This only makes sense when her primary concern is to make her life easier. But that was only one of three constraints.
Reducing the complexity of the problem in order to reduce the complexity of the solution is ludicrous because that requires the business problem to be retarded, too (you cannot sell your kids for experiments/you cannot refuse to provide your customers with what they want). Businesses that do not meet their customer’s expectations go bankrupt. Yet, attacking the complexity of a problem is common within companies in the financial sector. The strategy of retarding internal customer demands is at first ‘unconscious incompetence’. Later, upon realising a lack of abilities, as granny and I did with the washing machine and Java, it is a strategy born of consciousness. The financial engineer, his team and organisation at this point can make a choice. They can choose to accept the situation, choose to improve their technical and managerial capabilities. In doing so, they can align their private goals with that of their internal client teams’ (and therefore the organisation’s). Alternatively, they can choose to retard the demands of the customers (and therefore the organisation), choose to deny that their problems have evolved and so too must their solutions.
It is fair to call those who aim to retard demands ‘retromovement activists’. (Scharmer, p. 5) The retromovement activist, instead of adapting to the world, wants the world to conform to their standards, to their skill levels, which are based on the fuzzy memory of times gone by or on an emotional need for chaos. In his 2007 book, Theory U, Scharmer offers a neat taxonomy.
Defending the status quo is also a management strategy. It is sometimes, such as when running a factory, useful. For financial companies, it’s pretty useless. Regulations change. The amount of numbers to crunch are always increasing, which is why we start with Excel, migrate to Access with a C♯ client and over time end up with optimised databases and super computers. If a retromover runs from his problems into the jaws of a waiting tiger, a status quo defender freezes in the lights of a truck and gets squished. Human beings are never so obviously suicidal as when they are status quoing. Easter Island, in the Pacific Ocean, is famous for its gigantic stone heads and its extinct former inhabitants. They chopped down every tree in order to maintain their ways of life. More of the same. More houses. More statues. More bonfires. More barbecues. No more trees. No more life. Denial of the situation was followed by despair. In his book, Collapse, Jared Diamond says,
I have often asked myself, ‘What did the Easter Islander who cut down the last palm tree say while he was doing it?’ Like modern loggers, did he shout, ‘Jobs, not trees!’? Or: ‘Technology will solve our problems, never fear, we will find a substitute for wood?’ (Diamond, p. 114)
Technology wouldn’t have saved the people of the island like it never saves troubled engineering teams. The problem of sustainability is always a people problem. Always a management problem (learning Java was a people problem too, I had to motivate myself, had to manage the change process). Like the inhabitants of Easter Island, more-of-the-same companies die. Or, in a Hobbesian way, they muddle on, the life expectancies of its employees: ‘solitary, poor, nasty, brutish and short’. (Hobbes, 1651)
A healthy organisation
comprises three systems: a tasks system or system of work roles designed to achieve the tasks an organisation is in business to perform; a sentient system, the system where the employees’ human needs for affiliation and identity are met; and an overarching management system that manages the relations between the two. (Hartel et al., p. 336)
I.e. a healthy organisation manages both the work the company performs and the people doing the work. Messing about with the complexity of tasks messes about with the organisation’s ability to perform. An organisation is not healthy, then, when one or more of the three systems are badly functioning. If the company is making huge profit but has high attrition, sickness or depression rates, it is succeeding to run as a business to the detriment of its staff.There are a number of case studies published, and I have some clients too, where this is true. Famously, Electronic Arts, back in 2004, were the subject of a programmer’s spouse’s ire. (Ea_Spouse, 2004) If, which is often the case with rich finance companies, it’s the other way round, the employees are having a great time but the company is failing to grow, evolve, and meet new challenges, then it is unhealthy. In this case, there is a good chance that retromovers or status quoers are in charge. Of course, since a company has to have some people who care about it, then I don’t think it’s possible to have a happy workforce and be failing as a company. Therefore, if a company is failing to meet its business goals then it can’t have a satisfied workforce by definition. (This is universally the case in the failing businesses I’ve helped.) If Granny didn’t do the wash and enjoyed kicking back for a week, she wouldn’t be enjoying herself for too long before all hell broke loose when the kids went to the sock draw and found it empty; you can’t have a failing business and happy staff/customers. You can’t have a happy staff without a system of management. You can’t have a system of management when a combination of retro-movers and status quoers run the company.
*
Breaking out of such patterns starts with awareness, which drives intention which in turn drives action. Deming said,
Quality begins with intent, which is fixed by management. The intent must be translated by engineers and others into plans, specifications, tests, production. (p. 5, Deming, 1983)
In Quality Software Management, Gerald Weinberg discusses patterns of organisations. The first three are the most interesting. Pattern 0, Oblivious, you will recognise from my earlier examples. Weinberg makes a telling point, ‘In Pattern 0, there is no software development organisation separate from the software user’. (Weinberg, p. 23) This is me and my Java Virtual Machine, this is a trader and Excel, this is the scientist and his command prompt. This is the problem provider and the problem solver being the same person or entity. This is the start-up situation that retromovers will pine for.
Weinberg’s Pattern 1, Variable, is more familiar. Pattern 1 emerges from pattern 0 when ‘problem solvers become aware, rightly or wrongly, that they are out of their depth’. (Weinberg, p. 24) When the problem solver is out of their depth, they ask for help. A separation occurs. The process of developing software becomes impossible to ignore. The interim solution back at 161 little Beverley High Road was that my mother got to stay off school and help with the wash. The process of getting the wash into the machine and through the mangler was impossible to ignore. A separation of tasks - mum loading and unloading, granny sorting - occurred.
Pattern 2, Managed, is when a company returns to oblivion, not in regards to producing software, they get that by now, but in regards to management. Pattern 2 represents an organisation’s fledgling attempts to get the development of software under control.
Pattern 3, Steering, is were pattern 2 companies should be aiming for. In pattern 3, solutions emerge and processes are steered to solve problems. Pattern 3 is staffed, by definition, by managers who 1) know what they are doing and 2) whose goals are aligned with that of the organisation’s. I.e. they want the company to succeed and are skilled and emotionally mature enough to see the bigger picture. Pattern 3 is the end-state that retromovers and status quo defenders do not want the company to get to.
My friend Mike and I play touch-rugby. He recently organised a league. He had to schedule the games and the referees. Eight teams, four referees per evening from a choice of five. The fifth, unused referee, is the evening’s coordinator. Simple. Mike is not a consumer of software as such, he’s a ‘business man’ with a problem. Up until that point in his life as the touch-rugby organiser, he’d never needed much more than a pen, some paper, and an email account to function. Until this year, Touch Rugby Netherlands was a pattern 0 software organisation. When Mike sat down to write the schedule, rightly or wrongly, he became aware that he was out of his depth. He asked his wife to help. Mike became, and became aware that he was, a problem provider. His wife, a financial engineer, became the problem solver. A separation occurred. They had to communicate. I don’t know how smoothly their process of development went, but a process was followed. I can imagine Mike’s wife getting annoyed that he wasn’t available to explain the constraints of the system - games per night, number of teams, number of referees, etc. I can imagine him getting annoyed that she didn’t ‘get’ that the last week was a knock out week.
The Touch Rugby Association is now a pattern 1, Variable, software organisation. The software produced is based on Mike’s wife’s skills. Should she be unavailable for the next little programming job and I have to do it, the results will vary because she and I vary. (She’s miles better with scheduling algorithms.) Mike and his wife - semi-oblivious to it - have:
Weinberg says that, ‘Because this[, pattern 1,] is the first pattern to have this separation of responsibility for quality, it’s the first pattern in which blaming appears to be a substantial software development activity’. (Weinberg, p. 24) It makes sense that once a separation between a problem solver and provider occurs, the fights will be not far behind. It’s not hard for me, since I know them both, to imagine my mother and grandmother bickering about who did what to the washing machine. It’s not difficult to imagine the rugby organisation trying to schedule a 20 team tournament and getting in a muddle. And it’s not necessary to imagine financial teams, because I see them all the time, fighting with each other over VBA solutions vs. database backed .Net web applications. I do not have to try hard to recall pre-novices, the unconscious incompetent, dismiss the basics of software development, such as source control, because it ‘wasn’t needed before’. And the classic response from problem providers in a level 1 company, ‘I don’t care how it’s developed, just make it work’, is present in every work journal I’ve ever kept.
A problem provider saying that they don’t care about how a system is developed is oxymoronic: the quality of a solution will only be as good as the quality of the problem statement. (Not knowing this betrays the problem provider as unconsciously incompetent.) If a problem provider doesn’t care, the problem solver can’t get to the information they need to provide the solution. It’s worth invoking Cross and Dorst, who said,
It is widely accepted that creative design is not a matter of first fixing the problem and then searching for a satisfactory solution concept; instead it seems more to be a matter of developing and refining together both the formulation of the problem and ideas for its solution, with constant iterations of analysis, synthesis and evaluation processes between the two “spaces” - problem and solution. (In Brooks, 2010, p. 51)
It’s at best rude for a problem provider to say that they don’t care about the development process. But, more practically, it indicates that they do not have any skills in developing a problem or its solution. ‘Just getting it working’ is a strategy that works for small problems, the engineer can stumble upon answers, use his instincts, etc. For anything of any complexity, it will fail. This is how an engineer can go from being a hero to a villain; it appears as if they became useless overnight. They do not. Problems evolve faster than an organisation’s capabilities do. An engineer (or team of engineers) can appear to be useless, all of a sudden. But, this is the symptom of a wider evolution in complexity - a failing engineering team is nearly always a symptom of a failing organisation, just as a failing police force is a symptom of a failing society. The performance of an engineer will be, as a company evolves, highly variable (depending on the complexity of the problem and the engineers' skills). This variation becomes intolerable. ‘Someone -’, the collective mind thinks, ‘needs to manage these programmers’.
This demand for management, the demand to remove variation from the development process, is filled by anyone the organisation can find, usually:
The mistake pattern 2 companies make is that they assume anyone can manage, just as anyone can test a web-site (‘just click a few links’). Weinberg says,
Crosby [the author of Quality is Free] characterizes the managers in this pattern [, pattern 2,] as ‘beginning to recognize that quality management can help but is unwilling to devote time and money to make it happen.’ There are several reasons they don’t provide the money or time:
Internal consumers, the ones who were fed up with the variances in quality in pattern 1, now find there is a management layer between them as problem providers and the engineers who are still the problem solvers. The customers, when the organisation matured to pattern 2, gave up their direct access to the programmers. For them, the new situation seems worse than it was before. They want to go back.
The manager, if he was a subject matter expert, was respected. He is now derided, his inadequacies obvious. He wants to go back too.
For fledgling financial engineer teams, pattern 2 symptoms include:
Because of the turgid nature of pattern 2, sometimes the retromovement succeeds. (You can go back!) The engineer teams are disbanded, sent back to the trading rooms as quants. This is done in the name of customer service - hoo-ha! - or as a protest against ‘management’. (Smart people don’t need managing, you see?) If this happens, an oscillation occurs. The organisation swings between pattern 1 and pattern 2. The instability is reminiscent of an addiction. We know that frustration directed at difficult problems is really only self-hatred, my frustrations at Java taught us that. So, an oscillating organisation is caught up in a cycle of self-hatred, hope, disappointment, and a regression to bad habits followed by a new resolution to ‘manage them programmers’! (‘I’ll never drink again’!)
The trick is not to go backwards but to use pattern 2 to springboard to pattern 3, Steering. This can only happen when the financial engineering team or the wider organisation recognises this oscillation for the lunacy it is. It’s madness. But it’s the madness that one finds on the road to sanity. The road to pattern 3.
In pattern 3, it is fair to say, management is considered a development tool. (Weinberg, p. 29) In practical terms, this means managers should be experts in managing engineering teams and organisations; they don’t have to be subject matter experts. Time, energy and money should be invested into management. For example, simple plan-do-study-act cycles should be set up. The training needs of the financial engineers should be attended to. A dedication to studying cause and effect must be initiated. Coordination between the problem solvers and the problem providers is needed; blame storming becomes brain storming. The strong and well equipped manager does not, will not, and cannot be bullied into previously bad working habits. Because they have a good understanding of the software development process, they are able to steer the group. No subject matter expert in scheduling, vanilla options, trading, or the boss’ mate from school can do this (nor, by the way, can an expert programmer). Only a subject matter expert whose expertise is software development, not programming, has any chance of producing high value and high quality software, the type of software that needs many people to make it, that needs coordination between problem providers and solvers.
Pattern 2, because pattern 1 is so easy to diagnose as deficient, is very easy to get into but very hard to get out of. Tension always exists between those who see the path out to pattern 3 and the retroactivists who would like to return to simpler, pattern 1, times. It’s this tension, and the oscillations in behaviour it creates, that sets a system of work that, in my experience, can last until the company ceases to exist. Pattern 2 teams, who periodically lapse into pattern 1, can seemingly live forever. Pattern 2 teams persist by utilising energetic, often young, staff members, burning them out, and then replacing them. Despite how morally repugnant such a system of work is, this is not a driver for many companies. This may not be surprising. Senge reminds us that complexity is intimidating, that it
can easily undermine confidence and responsibility - as is the frequent refrain, “It’s all too complex for me,” or “There’s nothing I can do. It’s the system”. (Senge, p. 69)
But Senge also provides an antidote. Simplification of the system is not the key to returned confidence and responsibility; a simpler system of work could not have got through granny’s ten child wash pile. Only understanding the new system restored a sense of confidence, which in turned created a greater ability to respond to the demands placed upon her. Insanity, in the face of very difficult problems, is therefore understandable, but not forgivable.
I have no real tips other than the obvious.
Further Reading
Arguably, the real purpose of philosophy is to relieve suffering via understanding. That, in many ways, is the real purpose of studying engineering and engineering management. In regards to this paper, there are a number of things worth considering. Firstly, we need to know what results leaders produce. The internet is full of case studies, as are the book shelves. Secondly, we need to understand the processes that bring about such results. Last year I enjoyed The Accountable Leader, The Five Dysfunctions of a Team, and Kotter’s Leading Change is very good. Finally, we need to understand what drives leaders. (Scharmer, p. 7) This is more interesting terrain. Daniel Goleman’s Emotional Intelligence and related books are worth looking at. Collins’ Good to Great is worth a look because he talks about James Stockdale, a survivor of a prison camp. This gives insight into the (seemingly contradictory) survival instincts of the leader. Viktor Frankl’s Man’s Search For Meaning again gives insight into the mind (remember that the leader’s internal state will be all over the team or company he manages). Frankl also discusses aggression, depression and addiction as symptoms to lack of meaning. In pattern 2 organisations, a lot of work is meaningless and conformity driven (because the incompetent manager hands out tasks and destroys rhythm). As an exercise, it’s worth cross referencing any pattern 2 companies you know with Frankl’s ideas. Bullying, coercion, and outbursts are aggression; sick leave, depression; addiction, coffee and cigarettes.
Weinberg’s Quality Software Management: Volume 1 Systems Thinking is very good. Inspirational. It should be on all engineering manager’s desks.
Acknowledgments
I have used Weinberg’s taxonomy of software subcultures as the framework for the latter half of this essay. I am indebted to Weinberg directly via his work and indirectly via the effects he has had on the wider engineering community. Weinberg leant heavily on Crosby, and so indirectly I owe him, too. I borrowed Scharmer’s term ‘retromovement activist’, which I think is brilliant. That is covered in Theory U. The opening quote is from James Baldwin, whose clarity as a writer and refusal to sugar coat his message is inspiring. All software and financial engineers would benefit from a slice of ‘Nobody Knows My Name’, which is collected in Vintage Baldwin.
Amy Chiu for helping me with Chinese web-sites I couldn’t decipher.
And last, but not least, all the financial engineers who emailed me, read drafts of this essay, who inspired me to post it, and who are battling daily with the maturation of their organisations.
References
Baldwin, J. (2004) Vintage Baldwin, Vintage.
Brooks, F. P. (2010) The Design of Design, Addison-Wesley.
Collins, J. (2001) Good to Great, HarperBusiness.
Crosby, P. B. (1980) Quality is Free, Signet.
Deming, W.E. (1983) Out of the Crisis, MIT.
Dive, B. (2008) The Accountable Leader, Kogan Page.
Diamond, J. (2005) Collapse, Penguin.
Ea_Spouse. ([2004] 2005) ‘EA: The Human Story’ in Spolsky, The Best Software Writing, Apress. Available from: http://ea-spouse.livejournal.com/274.html.
Frankl, V. E. (2006) Man’s Search for Meaning, Beacon.
Hobbes, T. (2003 [1651]) Leviathan. Available from: http://www.gutenberg.org/catalog/world/readfile?fk_files=1319137
Goldacre, B. (2010) ‘When the scientific evidence is unwelcome, people try to reason it away’, Guardian, 4 July. Available from: http://www.guardian.co.uk/commentisfree/2010/jul/03/confirmation-bias-scientific-evidence.
Goleman, D. (2006) Emotional Intelligence, Bantham.
Kahane, A. (2007 [2004]) Solving Tough Problems, Berrett-Koehler.
Kotter, J. P. (1996) Leading Change, Harvard Business Press.
Lencioni, P. M. (2008) The Five Dysfunctions of a Team, Wiley.
Nørretranders, T. (1998 [1991]) The User Illusion, Penguin.
Pizer and Hartel, ‘For Better or For Worse: Organizational Culture and Emotions’ in Hartel et al., Emotions in Organisational Behaviour. Lawrence Erlbaum, 2009.
Scharmer, O. C. (2007) Theory U, SOL (Society for Organizational Learning).
Senge, P. (2006 [1999]) The Fifth Discipline, Doubleday.
Tiley, P. (1985) Cloud Warrior (The Amtrak Wars, Book 1), Baen.
Weinberg, G. M. (1992) Quality Software Management, Volume 1: Systems Thinking, Dorset House.
Wells, H.G. ((2005) [1898]) War of the Worlds. Penguin. Available from: http://www.scribd.com/doc/85771/War-of-the-Worlds-by-H-G-Wells.
Internet Resources
Maslov's Four Stages of Competence available from: http://en.wikipedia.org/wiki/Four_stages_of_competence.
Gerald Weinberg, http://www.geraldmweinberg.com/Site/Home.html.
| < Prev |
|---|
Comments
What would you advise for engineers trying to encourage their managers to face reality and accept that change is necessary?
I have heard a related question being asked often during conferences: "How do I convince my organisation to go agile?" The answer often is, if an organisation doesn't 'get it' themselves, is to try and initiate something small and relatively unimportant for the company, so that you will have some freedom to do what you want. If successful, managers will want to latch on to it and, if possible, claim the success as their own. Instead of fighting this, perhaps you should use this impulse to your advantage: find some measures of success on your pilot project (say, number of reported bugs down dramatically, improved customer satisfaction, happier engineers) and bring it to the management's attention. If they favour their own methods over yours even in the face concrete evidence, the solution may be beyond your control.
You sound frustrated... Software is a balancing act between engineers and managers taking responsibility for their jobs. When we are all doing what we do best, whatever that is, then there should be no problems. If your colleagues, engineers or managers, don’t agree with this, you have a problem. But you can’t change them, you can change yourself, such as the way you communicate, the talks you give, the projects you run. As your credibility goes up so will your ability to influence. Tracking problems, opening up communication channels, etc.. It’s a long, hard process, but it starts with you.
The other thing about change is this: does your company need to change? Are the top bosses happy with the managers? Does the company make money? Does it have happy people? If so, then why should they change? If they don’t care, why do you? And if they don’t care, why are you staying? Do you own the company? Is your frustration really self-frustration? Remember the essay and the mangler and 20 GOTO 10. All you can do is work on yourself, become a true autodidact, get your credibility up, and patiently work with your managers if they want to improve. If they don’t, quit, have a party, wish them well, and move on. You are not solely responsible for the company you work for but you are solely responsible for your actions.
Consider the healthy organisation above and what they do at TW: http://martinfowler.com/bliki/ThreePillars.html
An earlier version of the essay appeared in his Oct 2000 CACM Column, and a summary can be found at http://www.corvusintl.com/CACM002-5OI.htm
RSS feed for comments to this post