Calculated Risk: Blog 3
When litigators advise on the merits of a case, they tend to use the same set of phrases: a strong case, a reasonably arguable claim, good prospects of success, more likely than not.
Superficially, these familiar phrases look and feel like one vocabulary for the same thing. Considered more carefully through the lens of decision science, these phrases capture the output of two distinct cognitive operations: evaluating and predicting.
Conflating these two operations threatens accuracy in assessing litigation risk – a strong-looking case may be mistaken for a very likely win – and all of the subsequent strategic decisions around settlement, reserves and funding are affected by that error.
Double Duty
In their seminal work, Kahneman and Tversky identified two distinct cognitive operations involved in judgment under uncertainty:
- Evaluating the input: forming a judgment about how good, strong or persuasive particular information is.
- Predicting the outcome: estimating how an as-yet-unknown event will actually turn out.
Evaluation reflects an analysis of what is in front of you now; prediction looks ahead to what is likely to happen in the future.
Close Cousins
While these two analyses are closely connected, there are instances where they can diverge.
In the context of recruitment, for example, assessing the quality of an applicant’s education and the extent of their experience is an evaluation; forecasting how well they will actually perform in the job is a prediction. The two judgments do not always align.
It follows from basic statistical logic that people should make more conservative predictions than corresponding evaluations of the same evidence. This is because prediction inherently involves more uncertainty than assessing evidence that already exists.
Kahneman and Tversky’s surprising finding was that people tend not to do this. Typically, predictions of future outcomes more or less match the strength of their evaluations.
To illustrate, in one of their studies, participants read a short description of a university student (e.g. intelligent, self-confident, well-read, hard-working, inquisitive).
One group was asked to rate the student against their peers (How impressive is this student’s profile relative to their peers?), while a second group was asked to predict their eventual grades (How will this student’s grades compare with those of their peers?).
Both judgments were given as percentile rankings on an identical scale, so they could be directly compared.
The two sets of judgments matched almost perfectly: a profile that was rated as more impressive than, say, 80% of students produced a prediction that the student would go on to outscore around 80% of them.
In effect, participants drew a straight line between the profile and expected future performance, as if the profile contained sufficient information to form an accurate prediction.
While factors like intelligence and curiosity are relevant to exam performance, many others play a role: sustained diligence, revision strategy, health, luck on the day. The cognitive error is not adjusting the prediction (in this case, down) sufficiently to take account of the uncertainty introduced by other factors.
This is referred to as under-regression. It has been replicated many times in the decades since and forms part of the cognitive foundations of the modern science of forecasting.
A Distinction with a Difference
The phrases that comprise a litigator’s “merits vocabulary” represent the outcome of these two distinct judgments:
- Phrases that evaluate the strength of the case or argument – e.g. “a reasonably arguable case that the clause was breached,” “a strong case on liability”.
- Phrases that predict the overall outcome of the case or argument – e.g. “good prospects”, “a 70% chance of success”.
This is not a distinction without a difference.
While these two judgments are closely connected – a stronger case is, all else being equal, a more likely win – they are not the same, and arriving at each one involves a different computation.
Critically, different people involved in litigation are often addressing different questions without knowing it.
The starting point for a litigator is usually the evaluation: how strong is this case?
The key question for a client, insurer or funder is more likely the prediction: what is the probability we’ll win?
More Than Words
A common complaint here is that “litigators hide behind words” and the popular fix is usually “give us a number instead”. This is too simplistic.
Converting “reasonably arguable” into a percentage chance of success assumes that the phrase constitutes a probability estimate when it doesn’t; it represents a statement about the strength of an argument.
The problem here is deeper than “words versus numbers”. Before converting any verbal judgment into numbers, there is a preliminary issue to be addressed: are we talking about the strength of the case or the likelihood of winning?
Separating these two questions is the first step in accurate risk assessment. Reaching the right number is a science in itself – and the subject of the rest of this series.
References
Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251.
This blog is the third in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Blog 2, Magnetic Numbers, looks at the anchoring bias in settlement discussions.