fnord

On the relationship between software quality, KPIs and leadership

Quality cannot be defined. If we do define it we are defining something less than Quality itself.*

When trying to produce Quality Software, we first need to have a better understanding of what we mean when we say “Quality”. The problem is “Quality” is something that can't actually be defined. Still, in a corporate context we are left with no choice but to deconstruct what we mean by “Quality” while remaining aware that by so doing we lose something of the very meaning we are looking for.

For example, we can look at the following attributes of the ever-elusive “Quality”:

• Stability
• Security
• Performance
• Relevant features
• etc...

What I mean … by the word ‘quality’ cannot be broken down into subjects and predicates. This is not because Quality is so mysterious but because Quality is so simple, immediate and direct.*

Having all the above attributes (no matter how many we end up defining) does not guarantee that the software we produce is Quality Software, but it's still a pretty good start. Thus, we start working towards these attributes.

Now the question becomes: “How do we know that we are making progress”? How can we tell whether our actions are actually making our software more stable, secure, performant etc.?

The usual answer would be that we need to further break down the attributes we are aiming for, while remaining aware that every time we do so we lose something of their meaning. After enough steps of “deconstructing” we end up with a set of measurements that serve as proxies of proxies – they approximate the attributes that approximate Quality. Things like defect counts, Cyclomatic Complexity, defects/LoC, uptime, user survey results etc...

If we want to be buzzword-compliant we may refer to these proxies-of-proxies as KPIs.

Given these KPIs it should be fairly straightforward to come up with goals and processes that should move them in the right direction, thereby guaranteeing we progress towards quality. Except that seems to rarely happen. There are several reasons why KPIs don't necessarily result in Quality Software – some of them may be:

• Time – it is very difficult to form long-term plans to improve KPIs that are measured frequently. One finds oneself having to constantly explain why the numbers haven't improved yet. Unfortunately, long-term plans tend to have the most impact.
• People – remember that the KPIs are proxies-of-proxies. People tend to focus on the immediate and tangible (numbers!) rather than on the broad and vague (“Quality”) and doubly so when said numbers have an impact on their pay. Thus most people will focus on changing the numbers and not on their underlying raison d'être(e.g. reduce defect count by not filing tickets for defects). Additionally, having KPIs makes failure to improve them very clear and obvious, and people are generally not very good at facing their own failures. This leads to much energy being expended on “explaining away” uncomfortable numbers. This, again, is more pronounced when said uncomfortable numbers have uncomfortable implications on pay, promotions etc.
• Organization – Every organization has a set of hidden KPIs that are implied by the behavior of its constituents. New arrivals quickly learn what is said to be valued versus what is actually valued. Another name for the set of implicit KPIs is “Organizational Culture”. An organization will always strive towards the implicit KPIs before even looking at the explicit ones.

A dilemma, which is Greek for “two premises,” has been likened to the front end of an angry and charging bull.*

We are faced with a conundrum – We need KPIs, because without them we don't know where we're going and how close we are to getting there. But we must also deal with Implicit KPIs, People focusing on numbers instead of intent and the tendency towards lack of long-term planning.

A more general way to phrase this dilemma is “how can we get people to do what's right instead of what's convenient”?

In our personal lives, the answer to this question is fairly obvious – culture, identity and values. I submit that the same answer holds in our working lives.

“Is it hard?” “Not if you have the right attitudes. It’s having the right attitudes that’s hard.”*

Happily, changing the culture, identity and values just so happens to be the main role of leaders (as opposed to “resource managers”). Thus, the best way for a leader to improve the quality of the software their team produces is to create a culture, identity and values that are congruent with producing Quality Software. The usual process for doing so is to define a vision that clearly states what the leader wants the organization to be. From this vision, a mission statement and a set of values can be derived.

Now all that's left is to change the current culture into the one the leader defined. As anyone who ever tried to drive culture changes in organizations knows, this is a matter of some difficulty.

Peace of mind produces right values; right values produce right thoughts. Right thoughts produce right actions and right actions produce work which will be a material reflection for others to see of the serenity at the center of it all.*

Following Boyd's OODA Loop model, the first step towards changing the culture is examining the current culture. As we just mentioned, in our context “culture” is synonymous with Implicit KPIs, which manifest through the current set of incentives and actions that prevail in the organization. John Cutler's table can be a great start in identifying the current culture.

Once the current incentives are identified, the next step (“Orient”) is to recognize the ones that conflict with the desired values – that is, to recognize cases where the Implicit KPIs contradict (to whatever degree) the explicit ones that we intend to set. We then need to formulate a plan on how to either protect the team from such contradictory incentives, or co-opt the incentives to our purposes (“Analyze”).

Finally, the plan needs to be put into action. At this point we announce the new set of values along with the corresponding set of incentives and then prove our sincerity by following the plan to countermand conflicting Implicit KPIs. This will likely require some level of conflict with superiors and colleagues. “Si vis pacem...”.

Once we have the right incentives in place, they should influence the team members into adopting the right values, which should manifest in actions that drive towards the quality attributes we defined, which would bring the team closer to producing Quality Software. This is also known as “leadership”.

---

• All quotes from Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance

Advanced Stats in Management

The concept of “Wins Above Replacement” exists in both basketball and baseball so-called advanced statistics. The meaning is very similar in both sports, but there is a subtle difference that got me thinking about how these stats would be applicable to managers (and really any kind of people who work as part of a team).

Baseball WAR/WARP

In baseball WAR/WARP is intended to measure the benefit a player brings to the team above a “replacement-level player: a player who may be added to the team for minimal cost and effort”. Of course, we all want to provide benefit to our team that is beyond “replacement-level”, but an interesting question is – what would be a “replacement-level manager”? What kind of manager “may be added to the team for minimal cost and effort”?

I submit that a replacement-level manager, going by the above definition, is a manager who follows all corporate policies to the letter; One who does as they are told, rarely refuses a request and performs exactly the amount of personal development required by HR. Such managers are very easy to come by – this is the default operating mode of anyone who was recently promoted to a management role and lacks confidence in their own abilities.

Thus the question: What do I, as a manager, bring to the team beyond doing as I'm told, distributing work to my team members and providing a semblance of personal development?

Basketball VORP/WAR

In basketball there are two different “above replacement” statistics, but they are both based on so-called box-score numbers – things like points scored, rebounds grabbed or shots blocked. Both VORP and WAR are based on formulas (BPM and RPM, respectively) that take all box-score numbers and outputs a single value intended to represent the value a player adds to the team in terms of additional points scored or wins obtained. Both formulas also define a “replacement-level” value and thus, naturally, a “value above replacement”.

So the question here is: What would be the “box-score” for a manager? And how would the different values be weighted to provide a single “above replacement” number?

On Communication Styles and the OSI Model

All real-world communication channels are noisy.

This is hardly ground-breaking truth, and seems obvious to pretty much anyone who ever studied or used any kind of communication channel. Yet I find that this principle can be applied to interpersonal communications and perhaps provide some insight.

First, let's define our terms. For our purposes, “Communication” means conveying and idea such that it leaves one mind and reaches another mind intact. “Channel” is a physical medium through which communication has to take place, pending the discovery of telepathy. “Noise” is any property of the channel that hinders communication.

Now, a very useful model when discussing communications is the OSI Model. Without getting into details, in the OSI model communication takes place across layers, where each layer provides a higher level of abstraction. Going by this model, “noise” can take place at different levels – a noisy environment can make it difficult for another person to hear what we are saying (physical level), having to focus on multiple things can make it difficult for them to understand (higher level), etc...

Looking at human communications through this lens can help us make some other useful analogies. For example, one can think of Active Listening as a TCP-like mode of communication where, every few “packets” (i.e. sentences) the listener acknowledges the receipt of the information, and ensures they understood it correctly (by repeating a sort of “checksum” of the data). In fact, the listener verifies that successful communication did in fact take place.

But the most interesting thing, for me, is to consider what constitutes “noise” at the highest abstraction level of interpersonal communication. At this abstraction level, we discuss direct communication of ideas from one mind to another and ignore the whole vibrating-flabby-pieces-of-flesh-moving-air-towards-a-small-membrane thing (or more likely digits-typing-on-keyboard-generating-electrical-pulses-that-cause-pixels-to-light-up).

I submit that at the top-most abstraction level, “noise” is the listener's experience, prejudice, bias and ideas about language usage[1]. In short, who the listener is.

Our experiences and biases, our ideas about the day to day usage of words, all of these (and more!) will have an impact on how we interpret ideas that are conveyed to us. This, I believe, is obvious to anyone who has been a language-using human being for more than about five minutes.

In my experience the way people deal with the noisiness of this communication channel falls into one of two categories:

Static – have a preconceived estimate of the noise level and type and communicate based on this estimate without validating it. In other words, have a preconceived idea of what experience, bias and prejudice is “acceptable” (typically based on the speaker's own experience, bias and prejudice) and assume (or even insist) that the listener falls in this “acceptable” range. Dynamic – start out by assuming some noise level and type and correct based on feedback received. Usually the initial estimate of the noise is either “highest possible” or, like above some value based on the speaker's definition of “acceptable” noise level.

It is easy to see why the first category is so attractive – it takes much less effort and places the burden on somebody else. There is an additional, perhaps more subtle, benefit too – it works pretty well as long as one only communicates with people who have similar experiences, biases and prejudices. This means that for most people, this mode of communication is not only easier but that in their experience it has always worked (until the first time it doesn't).

The second category of “error correction” is more difficult to implement. In fact, it requires an additional communication channel – a sort of “meta-channel” in which communication takes place about the communication itself. It also requires constant translation and adjustment of one's ideas in order to adapt to one's current (and constantly evolving) understanding of how best to communicate them. This is difficult, especially if one has never used this style of communication before.

However, there is, in my eyes, a very compelling reason to adopt the second mode of “error correction”. The reason is that it's the only way in which communication (as defined above) can intentionally take place. If I say my goal is to communicate (rather than “to talk”) then it is my responsibility to do everything in my power to make sure communication does in fact take place – that is, that my ideas are conveyed clearly into the minds of the specific people I am communicating with. And that means taking into account their experiences and biases – in short, I have to consider who they are.

The people who insist on using dictionary definitions, on being judged by their intentions rather than by how their words are received, who call out their listeners for being “too sensitive” – these people either don't want to communicate, or they forget (or choose to ignore) that they are communicating over a noisy channel. The noise is other people.

=====

[1] “noise” is a problematic term because it carries negative connotation of something undesirable that should be gotten rid of. This is of course not the case for people's experiences and identities. The alternative of “filters” is also problematic because in suggests something artificial that can be removed at will. I am happy to receive suggestions for better terms.