The three kinds of judgement

[originally posted to BlogCisive]

To a first approximation, all deliberative judgements (i.e., those that turn on to-some-degree careful consideration of the relevant arguments) can be usefully sorted into three kinds.

These are the three Ds of judgement.

1. Decision

Decision is a matter of choosing from among options, particularly where those options are possible actions.  The question here is “What should I (we) do?”

2. Diagnosis

Diagnostic judgements concern what is going on.   The question is “What is happening?” or “What’s the situation?”  The term diagnosis has medical connotations, but here I’m widening its use to include various kinds of investigation, hypothesis testing,  and problem-solving.  All diagnostic judgements involve hypotheses (conjectures) as to what is actually happening.  A good example of diagnostic judgement in this sense is the assessment in intelligence analysis.

3. Deliberation

Deliberation is trying to determine the truth of some proposition by considering the arguments for or against it.  The question is “Is it true?”

Austhink has two products – Rationale, and bCisive.  Rationale, the argument mapping tool, supports deliberation.  bCisive, the business decision mapping tool, has been positioned as supporting decision.  We haven’t had a tool for diagnosis, and have tended to recommend that people wanting to make diagnostic judgements use some variant of the “Analysis of Competing Hypotheses” (ACH) method.

However, just as argument mapping supports deliberation, and business decision mapping supports decision, so “hypothesis mapping,” an alternative to ACH, supports diagnosis.  Further, hypothesis mapping is quite easily handled in bCisive as it stands. 

Austhink is currently working on a “Pro” version of bCisive which will include crucial features needed for supporting both deliberation and diagnosis. 

This means that one tool will help users map the thinking behind all three major kinds of deliberative judgement. 

The tool should be available in a few months.

Mapping Out the Options

[Draft of a piece written for an IT weekly magazine]

“In the affair of so much importance to you, wherein you ask my advice, I cannot for want of sufficient premises, advise you what to determine, but if you please I will tell you how.”

So began some of the best advice on decision making ever given. 

It was 1772, and the great Benjamin Franklin was advising his scientific colleague Joseph Priestley. But his advice is just as relevant today, when critical business and IT decisions must be made under conditions of great uncertainty and time pressure.  

Recently, his insights have been incorporated into software packages which can improve and accelerate organisational decisions, including IT decisions. 

His counsel, briefly, was take a sheet of paper, divide it into two columns, and write down all the advantages of a certain path of action in one column, and disadvantages in the other.  Then, by “cancelling out” items in one column with items in the other, assess which column is the more weighty.

Simple but powerful, Franklin’s “Moral Algebra” has given great service over the decades.  Research has shown that such methods reliably produce better decisions than ordinary unstructured deliberation.  Even Charles Darwin deployed the method, using it in deciding whether to get married.  Balancing considerations such as “terrible waste of time” on one hand with “object to be beloved and played with” on the other, he drew the clear conclusion that marrying was the right option, resulting in a long and happy union and thirteen children.

A method good enough for Franklin and Darwin is not to be dismissed lightly.  But we must also acknowledge that, in its classic form, it can’t do justice to many business decisions.

Most obviously, the Moral Algebra frames the problem as whether to undertake a particular action. But most decisions are not yes/no or go/no-go; rather, they involve choosing from a range of possible actions.  For example in a hiring situation, the decision problem is likely to be not whether to hire Jones but rather which of Jones, Jiminez or Jagonski to hire.  Moreover, the options may form a kind of hierarchy; at one level the decision is whether to hire an accountant or a tax lawyer; at the next level, if an accountant is to be hired, which one?

Another problem is that the method focuses on only one part of the decision problem, which is how to do the overall “weighing up” of the various considerations.  But in business decisions, much of the work goes into determining the validity or strength of those considerations in the first place.  A claimed advantage of hiring a tax lawyer is that she can manage certain difficult issues to do with complex tax structures.  But is this really true? Perhaps the issues are too complex for any one lawyer, and external advice would have to be obtained anyway. 

Recognizing that we need something more than a simple pro/con approach, many textbooks, professors and consultants will prescribe moving to some technical, usually quantitative methodology such as multi-attribute utility theory or decision analysis.  These certainly have their uses, but the reality is that most decision makers do not use these technical methods for the bulk of their decisions, even if they were taught how to use them in business school.  And those decision makers are not just being lazy; often these sorts of complex analytical tools just don’t get a grip on the distinctive texture of business decisions.  Real problems often can’t be reduced to numbers, algorithms and decision rules. 

There is an alternative, one that works better for a wide range of decisions, whether in IT, strategy, or even in one’s personal life. Rather than discarding Franklin’s method, we can extend and adapt it to handle more of the complexities of business decision making.

Business Decision Mapping is such an extension.  It preserves Franklin’s essential insight, that deciding usually involves weighing up of diverse, usually qualitative considerations.  But it elaborates his method to handle multiple options and lower-level arguments and evidence.  But it keeps the complexity under control by laying all this out in a special kind of diagram – a decision map.

Here, briefly, are the most basic steps in decision mapping. (1) First frame the decision problem using an open question.  Not “Should we hire Jones?” but rather “Who should we hire?” – or even better, “What should we do about our tax accounting needs?”

Put the question in a box, preferably with a Question icon so its status is always immediately apparent. 

(2) Canvass the major actions you might take in response to the problem.  Write these down in boxes, connected by arrows to the question.

For each action, write down the most salient pros and cons.  Draw arrows to the relevant option boxes.  (3) Consider the major arguments bearing on the pros and cons.  Add them to the map.  (4) Consider the detailed pieces of evidence supporting the major arguments.  Add them also.

(5) When this has been done as exhaustively and rigorously as circumstances allow, evaluate the evidence, arguments, and the pros and cons.  (6) Choose the most strongly supported options. In making these assessments, you might use Franklin’s “cancelling out” approach. 

Compared with standard informal deliberation (e.g., arguing around the boardroom table), business decision mapping takes some extra effort but offers many advantages. 

First, it improves the clarity and rigour of thinking behind the decision.  With the thinking laid out in front of us, we can more easily survey the full range of considerations and take proper account of them. 

Second, it improves collaboration.  A decision map is an easier way to communicate a complex structure of options, argument and evidence.  With better sharing, team time is spent more productively. 

Third, the decision mapping process automatically results in a concrete record of the thinking behind the decision.  This is useful if – as often happens – the decision needs to be revisited at some later point in time.  It also helps the decision makers be accountable.  Once a decision is made, things might still turn out badly for other reasons; but at least the decision maker can easily show that the decision was well-grounded at the time.

Business decision mapping can be done on paper, whiteboard, or computer screen, using markers or generic software packages.   However, like most things, it can be done better and faster with dedicated tools.   In recent years, dedicated decision mapping software has emerged, making creating, modifying and sharing of decision maps relatively simple and fast.   Decision mapping, supported by such software, deserves a place as a standard part of the toolkit of IT analysts and executives.

Decision at the center

Chandler is an attempt to re-invent the “personal information manager” (think: Outlook).  Dreaming in Code is a fascinating book about the Chandler journey.

The Chandler developers had to think long and hard about the nature of knowledge work.  On their website, on a page called Chandler Project Vision, they describe their Target Users:

They work closely with every member of their team, acting as a communication hub. They know how to ask the right questions to gather input and feedback. They identify problem areas, figure out when meetings need to happen, who needs to be there, what needs to be discussed, and then they facilitate the discussion to define concrete next actions and ultimately drive their team towards informed decisions. They take on the responsibility of defining realistic goals for their team and getting everyone pointed in the same direction to reach those goals.

On this description, Chandler’s target users are very closely related to the target users of bCisive, i.e. “business decision makers.”

Alongside the description of their target users, the page has a graphic:

This graphic of course is a direct play on the 19th century phrenological diagram:

The Chandler graphic both pays homage to, and satirizes, those early attempts to relate mental function to cerebral locations.

As with the phrenological diagrams, the Chandler graphic’s assignment of functions to locations is almost completely fanciful.  However there is one thing that is worth saving in it, albeit in a metaphorical interpretation – the locating of Decide! in the center, almost like the hub around which all other higher mental functions revolve.  (Note also that it is the only activity which gets an exclamation mark!)

Something similar is true of decision in knowledge work – it is the pivotal task.  Every other type of task can be seen as leading up to a decision, or following on from the making of a decision.

This central place is reflected in the structure of organisations – generally speaking, the higher one is in an organisation, the more one’s job is about making important decisions.

[Cross-posted to BlogCisive]

Everyweek decisions

In a nutshell: describes the distinctive character of decisions managers or executives must make on a weekly basis.

It is a curious fact about the English language that the second most common word (“of”) occurs about half as often as the most common word (“the”); the third most common word occurs 1/3 as often as the most common word; and this pattern holds pretty well for the first 1000 words or more.

This pattern, known as Zipf’s Law, or more generally a power law distribution, occurs for a suprisingly diverse range of phenomena, from cities (the bigger they get, the fewer there are of that size) to earthquakes to price variations.

I don’t know of any any measurements to verify this, but it seems plausible that such a pattern would hold for business decisions as well. These decisions can be rated in terms of their level of significance, ranging from the trivial or everyday (Where should we go for lunch? Use “Yours sincerely” or “Cheers”?) to the truly momentous (Should we pay bribes to Saddam Hussein to sell more wheat?). In a given period, the typical manager or executive would make a great many everyday decisions, very few momentous ones, and a generous handful of “mid-sized” decisions.

Making everyday decisions would generally take only moments or at most minutes; momentous decisions might take months or years. Somewhere in between are decisions that the typical decision maker would make regularly but not everyday; she might make a few a week, and each would be thought about over a period of days or weeks. We might call these “everyweek” decisions (as opposed to everyday decisions).

Examples might be:

• Who should we hire for this job? (Should we fire Jones?)
• Should we revamp the current website, commission a new one – or what?
• Should we pay for a booth at the XX tradeshow?
• What legal structure should our US subsidiary have?

Everyweek decision problems tend to have the following properties:

• They involve considering a number of options, and “weighing up” the considerations bearing on these options.
• The weighing-up is done intuitively, i.e., without using any strict rule or calculation.
• Each option has various pros and cons.
• The pros and cons are heterogeneous, i.e., very different in nature.
• The pros and cons are generally qualitative; it is impracticable or even nonsensical to put numbers on them.
• The pros and cons may need to be backed up by evidence.
• Indeed, the pros and cons are often disputable; we may need to consider the arguments for or against them, and the debate may get quite complex.
• Generally not all the options, or pros and cons, are immediately apparent; it will take effort to “uncover” or generate them.
• Everyweek decision problems are made under time pressure.
• The time pressure rules out doing lots of research or investigation; we have to “make do” what what knowledge or beliefs we have already, or can access quickly and easily
• Everyweek decisions are collaborative, i.e., there are multiple people involved in thinking through the decision.
• These people may be remote, i.e. far away and in a different time zone
• The decision maker is “accountable” for the decision.
• In particular, the decision maker may have to justify the decision, i.e., explain the thinking behind it

It seems that the great majority of the non-trivial decisions the typical business decision maker has to make are “everyweek” in scale and nature.

I’m hoping, in subsequent posts, to discuss a number of issues related to everyweek decisions, such as:

1. How everyweek decisions are generally made;
2. What problems or “pain points” there are in everyweek decision making;
3. The lack of good tools to support everyweek decision making
4. The direction we (Austhink) are taking in addressing these issues.

On Buying Cheese

The current issue of Choice Magazine (the Australian “Consumer Reports”) has a report on cheddar cheese.

They had five experts blindly rate 28 cheddar cheeses, ranging from your cloth- or wax-wrapped special deli cheddar at $50+ dollars per kilo down to the supermarket brands, sometimes less than $10 per kilo.

Eyeballing the results table, it seemed that price wasn’t a reliable guide to quality – some good cheeses were quite cheap and vice versa.

In the results table, they listed overall quality (score out of 20) and price per kg. They didn’t offer a “value for money” rating, so I copied the table into Excel and had it compute “value for money” as quality divided by price.

Now that the data was in Excel, we could probe a little further.

Turns out the correlation between quality and price was -.05. In other words, the quality of the cheese you buy, on average, has virtually nothing to do with price. If anything, as you go up in price, it gets worse.

Consequently, the correlation between quality and value for money was abysmal: -.8. In other words, on average, the more you pay, the more you’re getting ripped off.

Some cheeses had long names with lots of fancy-sounding words, such as “Devondale Special Reserve Premium Aged Vintage.” That must be a good cheese, right?

I used Excel to count the characters in a cheese’s name. Running the correlations showed that length of name bears little if any relation to price, quality, or value for money.

Conclusions: buying cheddar cheese is a lottery. If you haven’t tasted the cheeses, and are just trying to guess which ones are good, ignore price and fancy names; these have nothing to do with quality. If you want value for money, go for the cheaper cheese.

In short: when buying cheddar cheese in Australia, it just isn’t true that “you get what you pay for.”

PS – the cheese I’ll buy: South Cape Vintage Black Label. Nearly the top in quality, but only $15 a kilo.

Powered by ScribeFire.

Wise vs Smart

Paul Graham has an interesting post, Is It Worth Being Wise?, where he addresses what wisdom is, and how it differs from (“merely”) being smart or intelligent. He dismisses two supposedly-popular accounts:

• wisdom applies to human problems, and intelligence to abstract ones
• wisdom comes from experience, while intelligence is innate.

He suggests an alternative:

• “wise” means one has a high average outcome across all situations, and “smart” means one does spectacularly well in a few.

Without discussing each in detail, I’d say that we have here what JS Mill regarded as the most common situation, i.e., each of a set of apparently conflicting opinions has some part of the truth:

the conflicting doctrines, instead of being one true and the other false, share the truth between them; and the nonconforming opinion is needed to supply the remainder of the truth, of which the received doctrine embodies only a part. Popular opinions, on subjects not palpable to sense, are often true, but seldom or never the whole truth. They are a part of the truth; sometimes a greater, sometimes a smaller part, but exaggerated, distorted, and disjoined from the truths by which they ought to be accompanied and limited. (On Liberty, Ch.2)

Graham’s own suggestion seems to me not quite right, not least because it will be hard to distinguish being smart from being lucky, which on occasion makes some decision turn out spectacularly well. Indeed, in non-trivial domains (“subjects not palpable to sense”), luck is plausibly the operative factor in most decisions which turn out spectacularly well. Curiously, the words “luck” and “fortune” don’t appear in Graham’s essay.

Having had a career in academic philosophy, hanging out at times with people at the top levels, and now having spent a few years in business, I have my own hunch about the difference between wisdom and smartness. It is probably only part of the truth, but worth throwing into the mix.

Academic philosophers are often smart – sometimes spectacularly so – but are rarely wise. Hence, for example, the proliferation of inane opinions supported by powerful-seeming arguments. Meanwhile, in my limited experience of business, the proportion of people who strike me as very smart seems rather less than in academia – but at the same time there seems to be somewhat more wisdom around.

As the “CEO” of a small software enterprise, I’ve found that one of the most demanding parts of the job is being required so often to make decisions on matters where the consequences may be quite large, but there are multiple relevant factors, huge unknowns, and no reliable method of making a decision, at least in any reasonable time-frame.

Wisdom is being able to generally make good choices when confronted with such decisions.

Smartness, on the other hand, is being able to “figure things out.” A good example in our company is when Dan, our lead programmer, figured out the mathematical equations governing the elegant shape of the curvy lines in analysis mode. (I’m not suggesting that Dan isn’t wise, only that he is smart.)

In business decisions, very often you’re simply not able to figure things out. You don’t have the information, the time, or even a reliable method or tools. All you can really go by are your hunches, which are grounded in

• your own personal experience of similar situations
• your general background knowledge. For example, you may learn quite a lot from reading books, blogs, etc.
• relevant insight you can glean from discussions you have with colleagues, company directors, friends etc.

From these sources you get a vague “sense” of the situation which recommends (for better or worse) a way to go. Wise people have more experience (from which they’ve been able to learn); more background knowledge; good colleagues etc; and are able to exploit these resources by synthesizing relevant parts of it into a reasonable “take” on the problem.

Practically speaking, to go beyond just relying on the intuitive hunch based on a sense of the situation, there seem to be two strategies:

1. Apply some simple decision structuring framework – SWOT, multi-attribute utility matrix, etc.
2. Follow an “ideology” – a prior commitment to a strategy, goal or approach which simplifies and guides the decision.

“Smart” people will of course dismiss both of these as being intellectually childish. But people who are merely smart have the luxury of being smart because, generally, they don’t have to be wise. If they were required to be wise, my guess is they’d end up doing much the same thing, and wouldn’t be so quick to dismiss apparently simplistic approaches.

So in short, being wise is making a reasonable choice in an inherently speculative or vague situation, a choice guided by a good sense of the situation which in turn is based on the knowledge one has immediately at hand – prior experience, background knowledge, and the people around you.  Often, with decisions demanding wisdom, you’ll never really know ifyou were right, i.e., if one of the other choices would have been better.  Being smart is being able to apply general intellectual resources to “think through” a problem and arrive at a recognisably “right” answer.

Being wise is not intrinsically better than being smart. Both have their place, and of course one would like to be both.

Year in Ideas tidbits

Last week the New York Times published its “6th Annual Year in Ideas” in its Magazine.  81 bite-sized presentations of new ideas or trends; worth a read.  Trouble is, not much of what we read is remembered for very long.  Below I’ve listed the ones I’d like to be able to recall in a week or even a year’s time. 

• Empty-Stomach Intelligence.  Mice think better when they’re hungry.  Maybe we do too.  “Ghrelin” is the hormone at work.
• Eyes of Honesty. People are more virtuous when they’re being watched.  Even by obviously-fake eyes, such as on a poster.
• Hidden-Fee Economy.  People can be sorted into “sophisticates” who seek low up-front prices and avoid hidden fees (such as exorbitant mini-bar prices in hotels), and “myopes” (or “suckers”) who are constantly shelling out.  
• Homophily. Social websites are exploring ways to overcome “our inexorable tendency to link up with one another in ways that confirm rather than test our core beliefs.”
• Hyperopia - an excess of far-sightedness.  In the long term, ants tend to have more regrets than grasshoppers about how they spent their time.  This quirk of human psychology doesn’t mean you should just relax and indulge now though.   
• Low Starting Prices.  On Ebay, lower starting prices result in higher final prices – a kind of reverse anchoring effect.  This is because the anchoring effect is overcome by other effects.  A low start price results in many more bidders, enhancing the apparent value of the item, and exploiting people’s sense of commitment to create greater competition.
• Publication Probity.  The Journal of Spurious Correlations is devoted to publishing only negative results, aiming to counteract pervasive “publication bias” whereby positive results are published much more readily than negative, distorting our understanding of the world.
• Voting Booth Feng Shui.  How we vote is affected by where we vote.  People voting in schools are more likely to vote for state support for education.

In a corporate context…

 The note below was sent by a Rationale user in a major Australian corporation, who had received some coaching from Austhink’s Jane Lewis.

It arrived soon after I had finished the previous post suggesting that for more effective thinking, what we need is not more conscious thinking, or to leave things to our unconscious, but rather systematic processes supplemented by external resources which complement our inborn cognitive strengths. Case in point:

I have only been using Rationale for a very short time but, with your help, have been able to quickly learn the basics of argument mapping and been able to use argument maps to make important decisions and influence my management in their decision making. I know that I am developing a valuable skill.

My use of Rationale has been in 3 phases: organising arguments, interacting with peers and stakeholders to develop the argument, and presenting the argument to management. The first two tend to form an iterative process. Sometimes management feedback leads to another iteration series. I suspect there is scope for considerable value in doing more than I have done so far in the interaction phase.

The disciplines of capturing the elements of the argument; organising them in an argument map and then “abstracting up and filling down” leads me to a better and clearer understanding of my argument. It exposes shortcomings, occasionally leading to a re-evaluation of the value of a position. It also exposes openings into which new ideas can emerge.

The argument map is a wonderful communication tool – my peers can quickly see how I am thinking and provide relevant feedback.

Having assembled an argument, clarified it through organising it in a map, expanded it with new ideas and improved it with the feedback from my peers, I can present it to management. They can immediately see the entire argument on a single page. It is organised in terms of the key strategic drivers of the business (“abstracted up”) and the strengths and weaknesses of the argument are immediately apparent. They can quickly reach a position from which they can ask good questions or make valid criticisms. A key management skill is making good decisions quickly on minimal information – the argument map makes this easy.

Let me sketch out how argument mapping with Rationale has helped with a business decision of importance for my department.

The department owns several assets but one of these is not important to our core business objectives. Our company is doing well and a key goal for us is to keep focussed on work that delivers maximum value. Managing unimportant assets is a low-value activity but external constraints prevent simple disposal. I constructed an argument map in support of the proposition that we should trade this asset for an option over other more valuable assets.

Organising the map by abstracting up allowed me to focus on the key issues of COST, BUSINESS FOCUS, COMPETITIVE ADVANTAGE, RISK MANAGEMENT and OPTIONALITY. The arguments for and against the proposition were easily organised in terms of these themes with Rationale. Then I discovered a clear space in the map and immediately a whole new option for our business emerged. I realised that there was completely different way to acquire new options which leads to an elegant method for determining the value of those options. Valuing options is crucially important in our business and good valuations provide a key advantage in negotiations. Mapping the argument gave me a powerful new insight.

I described the proposition to my manager (without showing him the map) – his initial response was negative and he gave his reasons but the mapping process had prepared me. I was able to easily refute his reasoning and describe the value in my proposition. Within 5 minutes he was persuaded and enthusiastically in favour of the idea.

He took the proposition up to his manager and was asked to write a 4 page memo to support the proposition. My manager and I spent many hours honing the memo to present a clear argument. Once it was sent and read, the reply was a long set of questions and criticisms. It was clear that we had failed to communicate our ideas; senior management simply do not have the time to spend thinking through an argument – it is easier for them to send off their immediate impressions and move to the next issue.

I sent my manager my argument map – the entire argument presented clearly in an organised way on a single page. We spent a little time further refining the map and then he was able to present it to senior management over breakfast and see comprehension dawn!

Next week we have a meeting at which we hope to have our proposition signed off!

[Reproduced with permission of the author.]