All About That Base

Calculated Risk: Blog 4

When a litigator calculates prospects of success, the launch point for their thinking is usually the specifics of the particular case. How strong is the documentary evidence? How are the witnesses likely to behave? What’s the leaning of the judge or tribunal? Taking all of the relevant factors in the round, what do we think is the probability that we’ll win?

As a cognitive operation, calculating prospects of success is a forecast – a prediction about a future outcome. What professional forecasters do first, rather than start with the specifics of the case at hand, is ask: how often do cases like this one typically succeed?

In statistics, that number is known as the base rate: the frequency (the probability) of an outcome across comparable cases.

The Right Start

In the context of litigation risk, the way to use a base rate is to take it as a starting probability – the prior – before you consider your own case at all. If a quarter of claims of a particular type succeed at trial, the base rate is 25%, and 25% is then the starting point for the assessment. The specific facts then move you up or down from that number according to how strong and how reliable they are. Where you land after that adjustment is your final assessed prospects – the posterior. (Statistical aside: this sequence of prior to posterior is the logic of Bayes’ theorem, though Bayes’ arithmetic is not needed here.)

Occupational Hazard

The fact that litigators do not regularly consult base rates first when assessing prospects is not a surprise. They are not, by training, professional forecasters.

Also, failing to consider the base rate – base rate neglect – is one of the most reliable findings in research on human judgment under uncertainty. 

The problem with starting the analysis with the specifics, rather than from a prior, is that it can introduce inaccuracy into the final risk assessment. Where the base rate is low, for example, base rate neglect tends to produce over-optimistic analyses. 

Thirty Engineers, Seventy Lawyers

Base rate neglect was first systematically documented by Kahneman and Tversky in their 1973 paper “On the psychology of prediction”. 

In one study, participants were given short personality descriptions of several individuals they were told had been drawn at random from a pool of 100 professionals (e.g. Jack is 45, married with 4 children; he is generally conservative, careful and ambitious; he is uninterested in political and social issues and spends most of his free time on hobbies including home carpentry, sailing and mathematical puzzles).

Participants were told that the pool comprised either 70 engineers and 30 lawyers (high-engineer group; N=86) or 30 engineers and 70 lawyers (low-engineer group; N=85). They were then asked to estimate the probability that the individual was an engineer, on a scale from 0 to 100.

Both groups saw the identical descriptions; the only thing that differed between them was the frequency of engineers in the pool which, in this instance, tracked perfectly to % probability. The estimates of each group should have differed substantially, but they didn’t: mean estimates (of probably-an-engineer) were 55% in the high-engineer condition and 50% in the low-engineer condition – a statistically significant, but minimal, difference.

The base rate was even flagged incidentally within each question. Estimate responses were gathered by completing the following statement: “The probability that Jack is one of the 70 engineers in the sample of 100 is __%”. It was still largely ignored.

Nothing to Go On

Participants in the same study were also given a “null description”: “Suppose now that you are given no information whatsoever about an individual chosen at random from the sample”. 

Estimates for those test profiles tracked the odds exactly: 70% probably-an-engineer in the high-engineer condition, 30% in the low-engineer condition. 

So it isn’t the case that the participants were simply bad at probability. Given no individuating information, they used the base rate precisely as they should have. Neglecting prior odds in the original demonstration was a selective effect. 

Where specific information was available, people ignored the prior and focused instead on how closely the description matched the stereotype of ‘engineer’. 

There are circumstances in which people do generally use base rates reliably, particularly when they are expressed as plain frequencies (Koehler, 1996). But litigation risk, where a litigator has a specific file to evaluate, is precisely the type of condition where base rates are ignored. 

Received Wisdom

Historically, obtaining accurate and reliable data on base rates within litigation has been very difficult. 

Experienced litigators generally have a strong instinctive sense of the odds within certain types of claim: frustration of contract almost never succeeds; challenges to arbitral awards are seldom successful in arbitration-friendly jurisdictions; the English courts have all but stopped piercing the corporate veil.

Very few of those instinctive odds can be checked against a published figure. 

Some courts publish outcome data – the English Commercial Court, for instance, reports annual statistics on challenges to arbitral awards under the Arbitration Act 1996 – but the categories are broad. For frustration or for veil-piercing, for example, there are no equivalent numbers available from official public sources. 

What litigators have instead is a professionally inherited feeling for the odds, picked up from senior colleagues, war stories and cases everyone remembers losing. Often, this gut feel is broadly right. It is not, however, an objective measurement, and there is nothing to verify it against.

Identifying specific base rates is hard and, for many types of claim, that instinctive sense may be absent altogether. This is where litigation analytics have made a real difference, making base rate figures for different types of claim much more accessible. 

Covering the Bases

Every case is unique on its facts. But being unique on the facts does not release a case from the population of outcomes to which it belongs. Using that population outcome data is important – and it doesn’t require complex statistics. 

It is simply the practice of asking, before turning to the specific file, what usually happens in cases like this one? – and then adjusting from that answer according to the facts.

This raises a harder question: which population does your case actually belong to? That is the subject of the next blog.

References

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251.

Koehler, J. J. (1996). The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behavioral and Brain Sciences, 19(1), 1–17.

This blog is the fourth in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Our other blogs cover overconfidence, the anchoring bias and separating strength from prediction.