Calculated Risk: Blog 5

The previous blog in this series (All About That Base) made the case that calculations of litigation risk should consider the base rate – the proportion of comparable cases that succeed – before the specific facts of the case. 

This approach poses an immediate difficulty: how do we decide what is comparable to this case?

Which Pool?

A claim can belong to several populations simultaneously: 

and each population carries its own success rate.

There is no uniquely “correct” comparator in such complex environments and so no single true base rate. This is the reference class problem, a long-standing puzzle in the philosophy of probability (Hájek, 2007).

Better Than Nothing

The reference class problem presents a genuine difficulty, but it is not a reason to abandon base rates altogether.

Unquantified senses 

The alternative to using an imperfect base rate is not a statistically clean assessment of the merits: it is usually an implicit base rate. 

Most experienced litigators do have some rough sense of how often particular types of claim succeed. But that rough sense is exactly that – a loose range rather than a fixed number. 

Moreover, it isn’t entirely clear (even to the lawyer holding it) when or how that rough sense is being applied. 

Because instinctive base rate senses like this aren’t often articulated expressly, the specific case cannot be forensically compared against them. 

Relatedly, where two or more lawyers are advising on the same matter, each may be working from a different implicit number, and no-one ever discovers the mismatch.

No sense at all

Alternatively, ignoring base rates altogether – even a rough implicit one – means starting instead from the tacit assumption that the case will be decided only on its own facts.

That puts all the weight on the specifics of the case and none on the population. This is the least defensible starting point because it treats comparable cases as carrying no information about one another. 

Also, as with implicit base rate judgments, because the assumption is never stated explicitly, it cannot be challenged.

An explicit base rate, even an imperfect one, therefore has the advantages of being both quantified and visible. 

If the risk assessment records that “35% of these claims succeed at trial” as the base rate, a colleague can argue with the chosen reference class, with the pool of data used to produce it or with the sample size of comparable cases. A number held in someone’s head rather than said out loud can never be critiqued in this way.

Picking Your Pool

The reference class problem is universal. Anyone involved in predicting outcomes – doctors, insurers, forecasters, funders – faces the same issue, and none of them responds by ignoring base rates. 

Instead of trying to find the impossible one true number, they choose their reference class intentionally and transparently. The question to ask is: what is the most relevant and reliable comparator available, and how good is it?

Even the simple act of thinking through this question is likely to improve the quality of the litigation risk analysis. 

Narrow classes (e.g. this type of claim before this particular judge) are more relevant but noisier because they have fewer comparable cases. They therefore offer a less precise base rate percentage (small samples produce higher-variance estimates). 

Broad classes (e.g. all commercial claims) are more robust but less specific. Judgment is therefore required in determining the appropriate reference class of cases. The ideal is to choose the narrowest class that contains enough reliable cases and, better still, consider more than one.

If “termination disputes in long-term supply contracts” (subject matter) and “contract claims of this quantum” (quantum) both have base rate success figures around 30-40%, that is useful to know. If they diverge sharply (e.g. subject matter 15%, quantum 60%), at least one of those classes is the wrong comparator for your case. The expertise is in correctly distinguishing which. Luckily for lawyers, this is a legal judgment about what really drives outcomes in this kind of dispute, not a statistical one.

Fair Neglect

There are, of course, limits and exceptions to base rate methodology.

The unreliable or irrelevant comparator 

First, a base rate helps a risk assessment to the extent that it is reliable and relevant. 

An entirely unreliable or irrelevant base rate can be worse than none because that number may exert an unwarranted pull on the final probability assessment, as a result of the anchoring bias (see blog 2 in this series, Magnetic Numbers). 

A base rate from an entirely wrong population, or with too few cases, or from stale data, will anchor the risk assessment without actually informing it. 

The right discipline is therefore not “pick any base rate and always defer to it”. It is to find the most relevant, reliable base rate available and think critically about how to distinguish it, and how far to adjust from it. 

Discarding a poor reference class is not base rate neglect – it is good judgment (Welsh & Navarro, 2012).

The compelling exception 

Second, when the specific evidence is genuinely overwhelming and reliable, it can and should outweigh the prior base rate. 

Challenges to arbitral awards are a good example. Such challenges rarely succeed, but sometimes they do. Where the irregularity is flagrant and the resulting injustice obvious, the usual low base rate is properly displaced by the evidence. In that instance, the base rate isn’t wrong; it is comprehensively outweighed.

Those cases are rare. The far more common situation is that evidence which feels compelling is not, on examination, reliable enough to justify moving off the base rate.

Good Enough

The reference class will never be perfect, and it doesn’t need to be. What matters is that it is chosen carefully and stated explicitly. Writing a number down makes it possible for others to question it and suggest a better one, whereas a number that remains only in someone’s mind cannot be tested in the same way. 

Using the base rate requires fine legal judgment but not exotic statistics. Simply asking the question out loud forces the analysis of risk through some useful and important checks that would otherwise be skipped: which population, how good the data is, and how far this case really departs from it. 

References

Hájek, A. (2007). The reference class problem is your problem too. Synthese, 156(3), 563–585.

Welsh, M. B., & Navarro, D. J. (2012). Seeing is believing: Priors, trust, and base rate neglect. Organizational Behavior and Human Decision Processes, 119(1), 1–14.

This post is the fifth in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Blog 4, All About That Base, explains why considering the base rate improves final risk assessments.