Calculated Risk: Blog 5

The previous blog in this series (All About That Base) made the case that calculations of litigation risk should consider the base rate – the proportion of comparable cases that succeed – before the specific facts of the case. 

This approach poses an immediate difficulty: how do we decide what is comparable to this case?

Which Pool?

A claim can belong to several populations simultaneously: 

• all claims in the Commercial Court in England and Wales; 
• all claims with this type of termination provision;
• all claims before this particular judge;
• all claims of this particular type;

and each population carries its own success rate.

There is no uniquely “correct” comparator in such complex environments and so no single true base rate. This is the reference class problem, a long-standing puzzle in the philosophy of probability (Hájek, 2007).

Better Than Nothing

The reference class problem presents a genuine difficulty, but it is not a reason to abandon base rates altogether.

Unquantified senses 

The alternative to using an imperfect base rate is not a statistically clean assessment of the merits: it is usually an implicit base rate. 

Most experienced litigators do have some rough sense of how often particular types of claim succeed. But that rough sense is exactly that – a loose range rather than a fixed number. 

Moreover, it isn’t entirely clear (even to the lawyer holding it) when or how that rough sense is being applied. 

Because instinctive base rate senses like this aren’t often articulated expressly, the specific case cannot be forensically compared against them. 

Relatedly, where two or more lawyers are advising on the same matter, each may be working from a different implicit number, and no-one ever discovers the mismatch.

No sense at all

Alternatively, ignoring base rates altogether – even a rough implicit one – means starting instead from the tacit assumption that the case will be decided only on its own facts.

That puts all the weight on the specifics of the case and none on the population. This is the least defensible starting point because it treats comparable cases as carrying no information about one another. 

Also, as with implicit base rate judgments, because the assumption is never stated explicitly, it cannot be challenged.

An explicit base rate, even an imperfect one, therefore has the advantages of being both quantified and visible. 

If the risk assessment records that “35% of these claims succeed at trial” as the base rate, a colleague can argue with the chosen reference class, with the pool of data used to produce it or with the sample size of comparable cases. A number held in someone’s head rather than said out loud can never be critiqued in this way.

Picking Your Pool

The reference class problem is universal. Anyone involved in predicting outcomes – doctors, insurers, forecasters, funders – faces the same issue, and none of them responds by ignoring base rates. 

Instead of trying to find the impossible one true number, they choose their reference class intentionally and transparently. The question to ask is: what is the most relevant and reliable comparator available, and how good is it?

Even the simple act of thinking through this question is likely to improve the quality of the litigation risk analysis. 

Narrow classes (e.g. this type of claim before this particular judge) are more relevant but noisier because they have fewer comparable cases. They therefore offer a less precise base rate percentage (small samples produce higher-variance estimates). 

Broad classes (e.g. all commercial claims) are more robust but less specific. Judgment is therefore required in determining the appropriate reference class of cases. The ideal is to choose the narrowest class that contains enough reliable cases and, better still, consider more than one.

If “termination disputes in long-term supply contracts” (subject matter) and “contract claims of this quantum” (quantum) both have base rate success figures around 30-40%, that is useful to know. If they diverge sharply (e.g. subject matter 15%, quantum 60%), at least one of those classes is the wrong comparator for your case. The expertise is in correctly distinguishing which. Luckily for lawyers, this is a legal judgment about what really drives outcomes in this kind of dispute, not a statistical one.

Fair Neglect

There are, of course, limits and exceptions to base rate methodology.

The unreliable or irrelevant comparator 

First, a base rate helps a risk assessment to the extent that it is reliable and relevant. 

An entirely unreliable or irrelevant base rate can be worse than none because that number may exert an unwarranted pull on the final probability assessment, as a result of the anchoring bias (see blog 2 in this series, Magnetic Numbers). 

A base rate from an entirely wrong population, or with too few cases, or from stale data, will anchor the risk assessment without actually informing it. 

The right discipline is therefore not “pick any base rate and always defer to it”. It is to find the most relevant, reliable base rate available and think critically about how to distinguish it, and how far to adjust from it. 

Discarding a poor reference class is not base rate neglect – it is good judgment (Welsh & Navarro, 2012).

The compelling exception 

Second, when the specific evidence is genuinely overwhelming and reliable, it can and should outweigh the prior base rate. 

Challenges to arbitral awards are a good example. Such challenges rarely succeed, but sometimes they do. Where the irregularity is flagrant and the resulting injustice obvious, the usual low base rate is properly displaced by the evidence. In that instance, the base rate isn’t wrong; it is comprehensively outweighed.

Those cases are rare. The far more common situation is that evidence which feels compelling is not, on examination, reliable enough to justify moving off the base rate.

Good Enough

The reference class will never be perfect, and it doesn’t need to be. What matters is that it is chosen carefully and stated explicitly. Writing a number down makes it possible for others to question it and suggest a better one, whereas a number that remains only in someone’s mind cannot be tested in the same way. 

Using the base rate requires fine legal judgment but not exotic statistics. Simply asking the question out loud forces the analysis of risk through some useful and important checks that would otherwise be skipped: which population, how good the data is, and how far this case really departs from it. 

References

Hájek, A. (2007). The reference class problem is your problem too. Synthese, 156(3), 563–585.

Welsh, M. B., & Navarro, D. J. (2012). Seeing is believing: Priors, trust, and base rate neglect. Organizational Behavior and Human Decision Processes, 119(1), 1–14.

This post is the fifth in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Blog 4, All About That Base, explains why considering the base rate improves final risk assessments.

Calculated Risk: Blog 4

When a litigator calculates prospects of success, the launch point for their thinking is usually the specifics of the particular case. How strong is the documentary evidence? How are the witnesses likely to behave? What’s the leaning of the judge or tribunal? Taking all of the relevant factors in the round, what do we think is the probability that we’ll win?

As a cognitive operation, calculating prospects of success is a forecast – a prediction about a future outcome. What professional forecasters do first, rather than start with the specifics of the case at hand, is ask: how often do cases like this one typically succeed?

In statistics, that number is known as the base rate: the frequency (the probability) of an outcome across comparable cases.

The Right Start

In the context of litigation risk, the way to use a base rate is to take it as a starting probability – the prior – before you consider your own case at all. If a quarter of claims of a particular type succeed at trial, the base rate is 25%, and 25% is then the starting point for the assessment. The specific facts then move you up or down from that number according to how strong and how reliable they are. Where you land after that adjustment is your final assessed prospects – the posterior. (Statistical aside: this sequence of prior to posterior is the logic of Bayes’ theorem, though Bayes’ arithmetic is not needed here.)

Occupational Hazard

The fact that litigators do not regularly consult base rates first when assessing prospects is not a surprise. They are not, by training, professional forecasters.

Also, failing to consider the base rate – base rate neglect – is one of the most reliable findings in research on human judgment under uncertainty. 

The problem with starting the analysis with the specifics, rather than from a prior, is that it can introduce inaccuracy into the final risk assessment. Where the base rate is low, for example, base rate neglect tends to produce over-optimistic analyses. 

Thirty Engineers, Seventy Lawyers

Base rate neglect was first systematically documented by Kahneman and Tversky in their 1973 paper “On the psychology of prediction”. 

In one study, participants were given short personality descriptions of several individuals they were told had been drawn at random from a pool of 100 professionals (e.g. Jack is 45, married with 4 children; he is generally conservative, careful and ambitious; he is uninterested in political and social issues and spends most of his free time on hobbies including home carpentry, sailing and mathematical puzzles).

Participants were told that the pool comprised either 70 engineers and 30 lawyers (high-engineer group; N=86) or 30 engineers and 70 lawyers (low-engineer group; N=85). They were then asked to estimate the probability that the individual was an engineer, on a scale from 0 to 100.

Both groups saw the identical descriptions; the only thing that differed between them was the frequency of engineers in the pool which, in this instance, tracked perfectly to % probability. The estimates of each group should have differed substantially, but they didn’t: mean estimates (of probably-an-engineer) were 55% in the high-engineer condition and 50% in the low-engineer condition – a statistically significant, but minimal, difference.

The base rate was even flagged incidentally within each question. Estimate responses were gathered by completing the following statement: “The probability that Jack is one of the 70 engineers in the sample of 100 is __%”. It was still largely ignored.

Nothing to Go On

Participants in the same study were also given a “null description”: “Suppose now that you are given no information whatsoever about an individual chosen at random from the sample”. 

Estimates for those test profiles tracked the odds exactly: 70% probably-an-engineer in the high-engineer condition, 30% in the low-engineer condition. 

So it isn’t the case that the participants were simply bad at probability. Given no individuating information, they used the base rate precisely as they should have. Neglecting prior odds in the original demonstration was a selective effect. 

Where specific information was available, people ignored the prior and focused instead on how closely the description matched the stereotype of ‘engineer’. 

There are circumstances in which people do generally use base rates reliably, particularly when they are expressed as plain frequencies (Koehler, 1996). But litigation risk, where a litigator has a specific file to evaluate, is precisely the type of condition where base rates are ignored. 

Received Wisdom

Historically, obtaining accurate and reliable data on base rates within litigation has been very difficult. 

Experienced litigators generally have a strong instinctive sense of the odds within certain types of claim: frustration of contract almost never succeeds; challenges to arbitral awards are seldom successful in arbitration-friendly jurisdictions; the English courts have all but stopped piercing the corporate veil.

Very few of those instinctive odds can be checked against a published figure. 

Some courts publish outcome data – the English Commercial Court, for instance, reports annual statistics on challenges to arbitral awards under the Arbitration Act 1996 – but the categories are broad. For frustration or for veil-piercing, for example, there are no equivalent numbers available from official public sources. 

What litigators have instead is a professionally inherited feeling for the odds, picked up from senior colleagues, war stories and cases everyone remembers losing. Often, this gut feel is broadly right. It is not, however, an objective measurement, and there is nothing to verify it against.

Identifying specific base rates is hard and, for many types of claim, that instinctive sense may be absent altogether. This is where litigation analytics have made a real difference, making base rate figures for different types of claim much more accessible. 

Covering the Bases

Every case is unique on its facts. But being unique on the facts does not release a case from the population of outcomes to which it belongs. Using that population outcome data is important – and it doesn’t require complex statistics. 

It is simply the practice of asking, before turning to the specific file, what usually happens in cases like this one? – and then adjusting from that answer according to the facts.

This raises a harder question: which population does your case actually belong to? That is the subject of the next blog.

References

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251.

Koehler, J. J. (1996). The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behavioral and Brain Sciences, 19(1), 1–17.

This blog is the fourth in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Our other blogs cover overconfidence, the anchoring bias and separating strength from prediction.

Calculated Risk: Blog 3 

When litigators advise on the merits of a case, they tend to use the same set of phrases: a strong case, a reasonably arguable claim, good prospects of success, more likely than not. 

Superficially, these familiar phrases look and feel like one vocabulary for the same thing. Considered more carefully through the lens of decision science, these phrases capture the output of two distinct cognitive operations: evaluating and predicting. 

Conflating these two operations threatens accuracy in assessing litigation risk – a strong-looking case may be mistaken for a very likely win – and all of the subsequent strategic decisions around settlement, reserves and funding are affected by that error.

Double Duty

In their seminal work, Kahneman and Tversky identified two distinct cognitive operations involved in judgment under uncertainty:  

1. Evaluating the input: forming a judgment about how good, strong or persuasive particular information is. 
2. Predicting the outcome: estimating how an as-yet-unknown event will actually turn out. 

Evaluation reflects an analysis of what is in front of you now; prediction looks ahead to what is likely to happen in the future. 

Close Cousins 

While these two analyses are closely connected, there are instances where they can diverge. 

In the context of recruitment, for example, assessing the quality of an applicant’s education and the extent of their experience is an evaluation; forecasting how well they will actually perform in the job is a prediction. The two judgments do not always align.

It follows from basic statistical logic that people should make more conservative predictions than corresponding evaluations of the same evidence. This is because prediction inherently involves more uncertainty than assessing evidence that already exists. 

Kahneman and Tversky’s surprising finding was that people tend not to do this. Typically, predictions of future outcomes more or less match the strength of their evaluations. 

To illustrate, in one of their studies, participants read a short description of a university student (e.g. intelligent, self-confident, well-read, hard-working, inquisitive).

One group was asked to rate the student against their peers (How impressive is this student’s profile relative to their peers?), while a second group was asked to predict their eventual grades (How will this student’s grades compare with those of their peers?). 

Both judgments were given as percentile rankings on an identical scale, so they could be directly compared. 

The two sets of judgments matched almost perfectly: a profile that was rated as more impressive than, say, 80% of students produced a prediction that the student would go on to outscore around 80% of them. 

In effect, participants drew a straight line between the profile and expected future performance, as if the profile contained sufficient information to form an accurate prediction. 

While factors like intelligence and curiosity are relevant to exam performance, many others play a role: sustained diligence, revision strategy, health, luck on the day. The cognitive error is not adjusting the prediction (in this case, down) sufficiently to take account of the uncertainty introduced by other factors.

This is referred to as under-regression. It has been replicated many times in the decades since and forms part of the cognitive foundations of the modern science of forecasting. 

A Distinction with a Difference 

The phrases that comprise a litigator’s “merits vocabulary” represent the outcome of these two distinct judgments: 

1. Phrases that evaluate the strength of the case or argument – e.g. “a reasonably arguable case that the clause was breached,” “a strong case on liability”.
2. Phrases that predict the overall outcome of the case or argument – e.g. “good prospects”, “a 70% chance of success”. 

This is not a distinction without a difference. 

While these two judgments are closely connected – a stronger case is, all else being equal, a more likely win – they are not the same, and arriving at each one involves a different computation.

Critically, different people involved in litigation are often addressing different questions without knowing it. 

The starting point for a litigator is usually the evaluation: how strong is this case? 

The key question for a client, insurer or funder is more likely the prediction: what is the probability we’ll win? 

More Than Words

A common complaint here is that “litigators hide behind words” and the popular fix is usually “give us a number instead”. This is too simplistic. 

Converting “reasonably arguable” into a percentage chance of success assumes that the phrase constitutes a probability estimate when it doesn’t; it represents a statement about the strength of an argument. 

The problem here is deeper than “words versus numbers”. Before converting any verbal judgment into numbers, there is a preliminary issue to be addressed: are we talking about the strength of the case or the likelihood of winning? 

Separating these two questions is the first step in accurate risk assessment. Reaching the right number is a science in itself – and the subject of the rest of this series.

References 

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251.

This blog is the third in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Blog 2, Magnetic Numbers, looks at the anchoring bias in settlement discussions.

Calculated Risk: Blog 2

For a profession so deeply built on words, it seems strange that litigation always ultimately comes down to a number: What shall we settle for? What quantum did the court award?

These are the questions clients care about most. From the perspective of decision science, they are judgments made under uncertainty: numerical assessments based on incomplete, often contradictory evidence, and reached without knowing how the witnesses will perform, how the judge or tribunal will react, or which way the case will finally go. 

A great deal rides on getting such numbers roughly right. Anyone involved in litigation should therefore understand that the first number put on the table often shapes how every later one is judged, as a result of the anchoring bias.

First Impressions

Anchoring is a cognitive bias that operates on numerical judgments (i.e. any assessment whose answer is a number). It is a systematic error in which an initial figure pulls every subsequent number towards it, even when that figure is arbitrary, implausible or obviously irrelevant.

It was first demonstrated by Tversky and Kahneman in their seminal 1974 paper in Science. They asked participants to estimate various quantities expressed as percentages, such as the proportion of African countries in the United Nations, and to give each estimate by reference to a number that had been generated by a (rigged) wheel of fortune.

The random, irrelevant number swayed the estimates heavily: those who saw the wheel stop on 10 guessed that 25% of UN member states were African, while those who saw it stop on 65 guessed 45% (these are the median estimates). The wheel obviously carried no information, yet it shifted the answers by twenty percentage points.

While the true proportion in this instance is, of course, certain, few people actually know it. Most anchoring studies since – and especially those in the legal field – examine judgments that have no single right answer, such as the value of a claim or the length of a criminal sentence.

Across studies, the anchoring bias is strong and pervasive. It has been replicated reliably across students, mock jurors, senior lawyers and judges, and it affects the exact judgment that assessing litigation risk requires: putting a number to something fundamentally uncertain.

Opening Gambit

Most litigators are at least intuitively aware of anchoring in the context of settlement negotiations, where numbers are discussed explicitly. An offer of £2m can feel like a reasonable compromise rather than a large sum when it follows an opening proposal of £7m from the other side.

The research bears out this intuition. In one experiment, people acting as a hypothetical claimant were significantly more likely to accept a settlement offer of $12,000 when it followed a low opening bid of $2,000 than when it followed a high opening of $10,000: 63% would accept after the low opening, compared with 34% after the high one (Korobkin and Guthrie, 1994).

An opening offer may not bear any relation to the  true merits – though ideally it would – yet it still operates to set a psychological reference point against which subsequent numbers are judged. For anyone weighing a serious settlement offer, understanding numerical anchoring is a strategic advantage, not an academic curiosity.

Loaded

There are good reasons to think legal professionals would be, if not immune to anchoring, then far less susceptible to it.

Judges and tribunals are trained decision-makers, experienced at making the same judgments (damages awards, sentencing decisions) many times over, and motivated by their reputations to get them as nearly right as anyone can. If any group should be able to resist the anchoring bias, it is judges.

The evidence, however, suggests otherwise.

Anchoring in quantum awards

One study tested the presence of anchoring among a group of 167 US federal magistrate judges (Guthrie, Rachlinski and Wistrich, 2001).

Each judge read the same case study, describing a serious personal injury claim: a self-employed electrician left unable to use his legs after a delivery company’s truck ran into him when its brakes failed. Liability was clear on the facts; the plaintiff claimed damages for lost wages, hospitalisation and suffering, but had not put a figure on the claim.

One group of judges was simply asked what they would award in compensatory damages. A second group was first asked to rule on a hopeless motion to dismiss – which argued, meritlessly, that the claim did not clear the $75,000 minimum the court needed to hear it – and only then to set damages.

The judges in the second group were not fooled by the motion: only two of the 87 in that group granted it (and one of those said they wanted the claim amended to specify an amount). However, the $75,000 still pulled their ultimate damages awards down: judges who never saw the motion awarded around $1.25m on average; those who saw the $75,000 first went on to award around $882,000 – close to a third less.

Anchoring in sentencing decisions

A different study tested the effect with an anchor that was not merely irrelevant but plainly random.

Fifty-two junior lawyers from various German courts, all recently qualified and newly sitting as judges, were each given a case study describing a woman who had been caught shoplifting for the twelfth time. They were asked to throw a pair of dice and to treat the result as the prosecutor’s sentencing demand, then to consider whether the sentence should be higher or lower than that number, and finally to decide the sentence themselves, in months of probation.

Unknown to the participating lawyers, the dice were loaded, so that half of them rolled a three and half of them rolled a nine.

Those whose dice totalled nine recommended an average of 7.81 months as the sentence; those whose dice totalled three recommended an average of 5.28 (Englich, Mussweiler and Strack, 2006). A number which the participants had produced themselves, from a source they knew to be random, still moved their final sentence by roughly half.

For anyone valuing a claim, there are two main implications from this research:

• First, if professional judges cannot mentally disregard a number they know to be irrelevant, parties negotiating under commercial pressure are unlikely to manage it unaided.

• Second, the “value of a case” is not fixed. Settlements are considered against the sum a court would likely award, but that figure is itself movable by the numbers appearing around it.

Ultimately, no-one is immune from the anchoring bias, and figures may be more malleable than they appear. 

Reasonable Doubt

There are two qualifications worth flagging here.

While anchoring undoubtedly occurs, there is considerable doubt over its influence in any particular case. Real disputes usually contain too much complexity to draw a straight line from one number to one outcome.

Also, the effect works below the level of conscious awareness, so even the individual making the decision cannot reliably report its effect.

These are not reasons to dismiss anchoring from consideration: a cognitive bias this strong and this well-evidenced is one to plan for and strategise around in advance.

Anchors Away

Anchoring is difficult to resist, and the evidence that we can train ourselves out of it is limited and mixed. So what can be done?

A few simple practices are likely to mitigate its influence.

The most important is simply to remember that the first number on the table has more influence than it should, however implausible that number may be.

Beyond this: settle on your own number before you hear the other side’s, so you have a well-reasoned and independent reference point to defend; and, when a figure is put to you, pause and deliberately generate the reasons it could be wrong before you respond.

References

Englich, B., Mussweiler, T. & Strack, F. (2006). Playing dice with criminal sentences: The influence of irrelevant anchors on experts’ judicial decision making. Personality and Social Psychology Bulletin, 32(2), 188–200.

Guthrie, C., Rachlinski, J. J. & Wistrich, A. J. (2001). Inside the judicial mind. Cornell Law Review, 86, 777–830.

Korobkin, R. & Guthrie, C. (1994). Psychological barriers to litigation settlement: An experimental approach. Michigan Law Review, 93, 107–192.

Tversky, A. & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

This blog is the second in the Calculated Risk series, exploring the psychological and numerical concepts that shape how litigation risk is assessed, communicated and understood. Blog 1, The Confidence Gap, looks at overconfidence in quantifying risk.

Calculated Risk: Blog 1

“What are our chances?” 

The answer to this deceptively simple question does more to shape litigation strategy than almost any other. How the case is run, whether and when to settle, what provision is made in the accounts – all of these decisions follow from a litigator’s view on prospects of success. That is precisely why being forced to put pen to paper on it – and to commit to a single figure – produces such unique and focused anxiety.

It’s also one of the most uncomfortable for lawyers, because it requires converting a mass of legal and factual complexity into a quantified (i.e. numerical) assessment of risk. The instinct to avoid pinning down a single figure is strong. As a result, most lawyers facing this question articulate their advice in words rather than numbers with phrases such as “good prospects”, “a very strong likelihood” or “more likely than not”.

Yet when lawyers do form a view of their clients’ prospects, research suggests that they are systematically too confident in their estimates of success.

False Positives

Overconfidence is a cognitive bias where our subjective confidence in something exceeds its objective reality. The psychological literature distinguishes three distinct types (Moore & Healy, 2008): 

1. Overestimation – an absolute judgment where you think you are better than you are (e.g. your probability estimates of success are optimistic compared to your actual win rate across cases);
2. Overplacement – a comparative judgment with unique application in adversarial contexts, where you believe you are better than others (e.g. you and your opponent both privately rate your respective clients’ chances as >50%; you can’t both be right); and
3. Overprecision – a judgment about your own certainty, where you think you know more precisely than you do (e.g. you give a single point estimate with no range, or a range of outcomes that’s too narrow).

All species of overconfidence can relate to the same prediction, appearing in different contexts: overestimation is relevant when you look at win rates over time; overplacement becomes evident at the settlement table; and overprecision arises the moment you lose an “unlosable” case. 

Of these three variants however, overestimation is the one most commonly understood to reflect “overconfidence” in lawyers’ assessments of litigation risk. The ultimate issue is how accurate – or how “well calibrated” – lawyers’ judgments are on prospects of success. In other words, when a litigator says “this case has X% chance of winning” (in instances where numerical values are given), do they actually win roughly that proportion of the cases they run? If they consistently win less often than they predict, their judgment reflects overestimation (overconfidence). 

Reality Check 

In 2010, Professor Jane Goodman-Delahunty and colleagues published an empirical study on this exact question of calibration in merits assessments, in the journal Psychology, Public Policy, and Law. 

The researchers recruited 481 litigators from 44 states across the United States, representing clients in real civil and criminal cases (the sample comprised 70% civil and 30% criminal cases). Each lawyer selected a live case that they expected to resolve within roughly 6 to 12 months and specified a minimum goal: what would count as success in this case? 

In civil cases, this was typically a monetary target (a damages figure) for the plaintiff’s lawyer or a “not liable” judgment for the defendant’s lawyer. In criminal cases, goals ranged from acquittal to conviction of lesser included offences for defence lawyers, or focused on successful convictions for prosecution lawyers. Each lawyer then estimated the probability, from 0 to 100%, that they would achieve that goal or better.

The researchers followed up after each case resolved and compared a lawyer’s prediction against the actual outcome.

The headline finding was striking and clear: lawyers as a group were significantly overconfident in their predictions. Across the full sample:

• 44% of outcomes were less satisfactory than the minimum goal;
• 32% of final case outcomes matched the goal; and
• 24% exceeded the lawyers’ minimum goal. 

The 44% is the figure that matters: in almost half of all cases, lawyers fell short of the lowest bar they had set themselves for success. 

Definitely Maybe

The calibration data (i.e. how often lawyers at each confidence level actually achieved their stated goal) told a more detailed story. 

Lawyers in the middle confidence ranges (46%-65%) were reasonably well calibrated. However, lawyers expressing confidence above 65% became progressively overconfident, and those in the highest band (86-100%) achieved their goals only about 70% of the time. 

In practice then, the strongest expression of confidence in a case’s success is often a signal that the estimate should be treated with particular caution, rather than as evidence of predictive accuracy. 

(No) Wiser With Age 

Notably, calibration did not improve with years’ experience. In this sample, senior lawyers were no better forecasters than junior lawyers (lawyers ranged from 0 to 45 years of practice, with the average being 13.3 years of practice).

One likely explanation for this surprising finding is that litigation provides very poor feedback conditions for learning. 

Unlike poker players, for example, who make hundreds of similar bets and discover quickly whether or not they were right, lawyers operate in what decision scientists call a “wicked” learning environment: outcomes are slow, every case is unique, feedback is ambiguous and it’s easy to explain away a poor result without updating one’s confidence. 

It is worth noting that the study measured experience as years in practice rather than domain-specific caseload, so it is possible that specialist expertise matters in ways the data in this study could not capture.

Seeing Blind Spots 

This overconfidence finding in lawyers’ merits assessments is neither a fluke nor a mystery. It has identifiable causes which, while largely absent from legal training, are well understood in decision science. 

Base rate neglect – that is, assessing a case purely on its specific facts without asking what percentage of comparable cases actually succeed – is one contributing cause. Compound probability – the fact that a chain of individually strong probabilities (e.g. on jurisdiction, breach, causation, quantum) can multiply down to a surprisingly weak overall prospect – is another. 

Concepts like these directly explain why lawyers’ probability estimates tend to be higher than they should be. 

The question on every client’s lips – “What are our chances?” – is one of the least comfortable moments for a litigator, precisely because it is one of the most consequential for the litigation. It also sits outside most lawyers’ natural skill set: litigators work with words not numbers, and they rarely receive any formal training on how to quantify uncertainty. 

The concepts that could improve those judgments – some statistical, some drawn from decision science – are well established. This series will explain them.

References

Goodman-Delahunty, J., Granhag, P. A., Hartwig, M., & Loftus, E. F. (2010). Insightful or wishful: Lawyers’ ability to predict case outcomes. Psychology, Public Policy, and Law, 16(2), 133–157.

Moore, D. A., & Healy, P. J. (2008). The trouble with overconfidence. Psychological Review, 115(2), 502–517.

The way witness statements are prepared in the Business and Property Courts of England and Wales was substantially reformed by Practice Direction 57AC, introduced in April 2021. The Practice Direction was a serious attempt to address something that judges and psychologists had been saying for years: the process by which witnesses are prepared to give evidence in civil proceedings is liable to corrupt their recollection of relevant events.

A paper published in May this year in Civil Justice Quarterly by Mark Steele SC offers the first detailed empirical look at how the Practice Direction is working in practice. Steele interviewed 35 participants who are actively involved in litigation in the Business and Property Courts: 11 judges (predominantly Commercial Court judges, with an average of seven years on the bench), 19 solicitors (litigators practising in the Business and Property Courts, with an average of 22 years’ experience) and 5 counsel (four King’s Counsel and one junior barrister, with an average of 29 years’ experience). His findings suggest that, while PD 57AC has achieved meaningful change – witness statements are generally shorter and more focused on disputed facts – the witness preparation process continues to be vulnerable to contamination in ways the Practice Direction did not fully anticipate.

One finding in particular stands out, because it reveals something that only becomes visible when you look at the interview data through the lens of memory science.

The document problem 

One of the provisions of PD 57AC that appears, according to Steele’s interviews, to have had a real impact on practice requires solicitors to list any documents that have been used to refresh a witness’s recollection on important disputed matters of fact. According to the study, this transparency requirement has changed behaviour in litigation: solicitors reported being much more cautious about which documents they show a witness, precisely because listing them creates what several interviewees described as a “red flag” for the opposing side to try to exploit in cross-examination.

That strategic caution, albeit aimed at preventing a different risk, could be a positive development from the perspective of reducing memory contamination. The concern is what appears to have emerged in its place. 

Steele’s interviews reveal that some solicitors, rather than showing a witness a document they would then have to list, are instead conveying the contents of that document to the witness orally – describing what the document says, or framing questions that draw on its contents, without putting the document itself in front of the witness. As one solicitor put it when asked about this use of oral prompts drawn from documents: “I think that’s fair game to be honest”. Another said “I think people definitely use documents to prompt a witness without showing them the document”, by asking something like “is it possible at the meeting that something like this was discussed?”. 

It is not difficult to see how this practice would evolve naturally: solicitors are working within the system the Practice Direction has created, balancing its transparency requirements against what they see as their duty to test the evidence effectively and to prepare a witness properly for trial. The shift to oral prompts is, in many cases, a pragmatic response to that tension rather than a deliberate attempt to circumvent the rules – though it doesn’t sit easily with the solicitor’s duty not to lead or coach a witness, or with the Practice Direction’s overarching requirement to avoid influencing recollection. From the perspective of the Practice Direction’s transparency requirements, this “oral prompts” approach avoids the need to disclose the document in question. From the perspective of memory science, it is almost certainly worse.

Easier said than written

Several decades of research on memory contamination point clearly in one direction: spoken communication is generally more likely than written material to distort a witness’s recollection, particularly when it is interactive, repeated or delivered by someone the witness perceives as authoritative.

Several factors explain why this is the case. One is what psychologists call “source monitoring” – the process by which our brains keep track of where a memory came from. When a witness reads something in a document, their brain is more likely to retain a trace of the source: a sense that this information was encountered in writing, on a particular occasion. When the same information is conveyed in conversation, the source is less reliably encoded. This isn’t to say that we never remember who told us what – we often do. But the research suggests that the contextual markers attached to spoken information are, on average, weaker and more easily confused with those attached to the witness’s own first-hand experience of the events. Information received in conversation – particularly conversation with someone the witness perceives as knowledgeable and authoritative, such as a solicitor who has studied the case file and spoken to other witnesses – integrates more readily into what feels like the witness’s own recollection.

Source monitoring confusion is compounded by the dynamics of face-to-face interaction. A conversation with a solicitor is socially interactive. It involves credibility cues – tone, confidence, the implicit authority of someone with detailed knowledge of the case. It may also carry an implicit pressure towards agreement, and it adapts dynamically to the witness’s responses in ways that a static document cannot. Each of these features makes it more likely that the information conveyed will be absorbed into the witness’s memory and later recalled as though it were part of their original experience. In the language of the “misinformation effect” – one of the most well-replicated findings in memory research – spoken post-event information is a particularly potent source of contamination precisely because it is harder for the witness to identify, after the fact, as having come from someone else.

The rule-practice gap 

The result is that the Practice Direction, in restricting the use of documents and mandating transparency about their use, has inadvertently created conditions in which solicitors may shift towards a form of communication that is more contaminating and less transparent.

The Working Group that developed PD 57AC comprised judges and senior practitioners, and while the Practice Direction draws on psychological research, the process of translating that research into practical rules for witness preparation appears to have been undertaken without direct input from memory scientists. The overarching requirement that witness statements should be prepared so as “to avoid so far as possible any practice that might alter or influence the recollection of the witness” is, as Steele’s interviews reveal, a provision to which most solicitors have devoted little attention. As one counsel observed: “[W]e’ve all got Gestmin and Lord Leggatt’s observations in mind, but beyond that I think that the profession is pretty simplistic in its understanding”. Another solicitor was more blunt on the absence of legal education in this area: “[t]here is no training really on any of that. I suspect most litigators worth their salt will have a high level view…but in terms of detailed working or training, I don’t think that exists”.

The training gap the interviews identify is a specific one: not familiarity with the Practice Direction’s requirements, but understanding of the memory science that underpins them. A requirement to avoid influencing a witness’s recollection is of limited practical value if the people tasked with implementing it have not been equipped with an understanding of what influences recollection – for instance, that the way a question is framed can alter the detail a witness recalls, that pressing for specificity where memory has faded tends to produce reconstructed rather than retrieved information, or that simply repeating an interview can cause the witness to become more confident in a recollection that has not become more accurate.

The work of the Working Group was a serious and largely successful reform, as Steele’s research confirms. What the study also offers is a rare glimpse inside what one judge described as “a black box” – showing how solicitors are actually navigating the new system the Practice Direction has created, and where the gaps between intention and practice have developed. 

The science of memory is not intuitive, however, and its implications for witness preparation are not obvious. Steele himself proposes expanding the Practice Direction’s transparency requirements to cover oral prompts as well as documents. This would be a welcome development for those attempting to identify potential sources of contamination. But transparency itself does not prevent the underlying risk of contamination. Reducing the underlying contamination depends on understanding how it occurs – and adjusting practice accordingly.

It is worth noting that in criminal proceedings, police officers receive dedicated training in interview techniques designed to minimise the risk of contaminating witness memory, and even that training does not produce perfect results. The difficulty of the task is not a reason to leave litigation solicitors without training; if anything, it is a reason to invest in it, particularly where the outcome of the case may turn on the reliability of a witness’s recollection of events that occurred years before.

Eighty-nine percent of international arbitration stakeholders believe that ethnic diversity on the arbitral tribunal supports the perceived legitimacy of the system. Eighty percent believe it contributes to perceived fairness. And yet sixty-eight percent of those same stakeholders rate the tribunals they have recently encountered as lacking in ethnic diversity.

That gap – between what the arbitration community says it values and what it is actually experiencing – is the headline finding of a report produced by Cortex for the International Bar Association’s Arbitration Committee, published in May 2025 and recently recognised with Global Arbitration Review’s Pledge Award. The report was built on more than three years of work: a pilot interview study with senior in-house counsel from diverse backgrounds worldwide, followed by a large-scale survey of 305 arbitration stakeholders across 76 nationalities. The aim was to gather empirical data where previously there had been none – on how the arbitration community views ethnic diversity on tribunals and what impact it has on perceptions of legitimacy, fairness and the quality of arbitral decision-making.

The Pledge Award, given by Global Arbitration Review, recognises initiatives demonstrating outstanding effort to advance diversity of all forms in arbitration. The Pledge itself was launched in May 2016 by members of the arbitration community to increase the appointment of women as arbitrators, with the ultimate goal of full parity. Its reach has since broadened, with recent awards celebrating work across a range of dimensions of diversity in the field.

With the Pledge Award bringing renewed attention to the findings in the report on ethnic diversity, we take this opportunity to draw out some of the observations that give the data real nuance – not as a detailed summary of the report, but as a way of highlighting what the numbers alone do not fully convey.

The values-reality disconnect 

The headline numbers above are stark. The disconnect is not news to anyone who works in this field, but having the data – from a large sample, across 76 nationalities – gives definition and scale to what has until now been largely anecdotal.

What the data in this report establish is not that ethnic diversity matters – few in the field would dispute that – but the size of the distance between principle and practice. A near-universal majority endorses diversity’s value, but a clear majority reports seeing tribunals that fall short of it. This is not a handful of dissenting voices: drawn from 305 respondents across 76 nationalities, the gap holds across a broad international sample. It is one thing to suspect a shared frustration; it is another to measure it. Quantifying the disconnect is the foundation for everything that follows. Since the report’s publication, the IBA Arbitration Committee has been hosting a series of roundtable dialogues, in cities around the world, on the issues it raises.

230 ethnicities, 76 nationalities

One of the more interesting findings emerged not from the survey questions themselves but from the demographic data – the background information collected from respondents at the end of the survey. In particular, when respondents were asked to self-identify their ethnicity, 230 different ethnicities were recorded (among the 240 respondents who completed the question). This is more than three times the number of different nationalities (76) captured in the same dataset. Moreover, within these figures, 34 respondents identified as mixed ethnicity, compared to just 19 with multiple nationalities.

This incidental finding matters because most arbitral institutions currently track diversity by nationality alone. Nationality is a useful and pragmatic proxy, but the data in the report suggest that it captures only a fraction of the ethnic diversity – or lack of it – that stakeholders actually experience. If the ultimate goal is broader and more meaningful representation, the lens through which this topic is viewed may need to be wider than the passport.

Another notable observation in relation to this ethnicity self-identification metric is that not all respondents engaged with the question in the same way. Sixty-five respondents chose not to provide their ethnicity at all – significantly more than the number who declined to provide their nationality, age or gender. One respondent said they had never thought about it. Two reported their ethnicity as “Human”, and one specified “Regional”. The data cannot explain the reasons behind these responses, but taken together the pattern illustrates the complexity involved in discussing ethnic identity with an international professional audience.

Lost in translation 

The data added a different dimension when respondents described the consequences of tribunals lacking cultural awareness. Their concerns were not abstract. 

One interview participant explained that in their culture, written correspondence with the opposing party follows particular conventions. For example, even in the heat of a dispute, letters open and close with expressions of goodwill. When their side’s correspondence was put before a tribunal unfamiliar with this practice, the friendly language was misread. As the participant put it: “In one arbitration, the other side disputed the fact that we were raising claims in our written correspondence. We had to explain to the tribunal that this apparently friendly language is something you always include in formal correspondence as a matter of course”.

Witness testimony raised similar issues. Non-verbal communication – eye contact, body language, tone – is interpreted very differently across cultures. In some Asian cultures, for instance, avoiding direct eye contact is a sign of respect. A tribunal unfamiliar with this may read it as evasiveness or dishonesty. One respondent put it bluntly: “If a witness from Asia is humble or timid, a tribunal composed of western arbitrators may think ‘that witness is lying’ or ‘they don’t look confident or assertive enough’ based on their body language”.

Other examples included the way apologies function across cultures. In certain contexts, for instance, an apology is not intended (and is not understood) to imply an admission of liability – but it might be interpreted that way by arbitrators applying legal standards rooted in a different cultural tradition. 

Respondents also flagged behavioural or procedural differences that can easily be misconstrued – against the relevant party – by a tribunal with a different cultural sensibility. For example, ex parte communications with an arbitrator may be viewed as strictly off-limits by parties from some traditions but not others. The deference and protocol surrounding senior officials in Middle Eastern disputes may be seen as entirely ordinary in those cultures, but read as evasion or even bad faith by a tribunal applying a different set of cultural assumptions.

These are not peripheral concerns. They affect how tribunals assess credibility, interpret evidence and conduct proceedings.

Legitimate expectations

The data on legitimacy and fairness were clear in aggregate, but the open comments added a dimension that numbers alone cannot. One respondent described seeing a construction dispute in their region where the tribunal comprised three English law lawyers. For that respondent, the composition alone raised questions about whether the process could genuinely understand the local context. As they put it: “When I see a construction case, for example, where the arbitrators are three English lawyers but the project is in my region, that undermines legitimacy in my view”.

Another respondent stated that “diversity creates representation,” and that without it, parties from certain regions might be deterred from choosing arbitration as a dispute resolution mechanism altogether. In the context of investor-state arbitration specifically, one respondent noted that “the fact that arbitrators in ISDS are overwhelmingly European or North American is causing a lot of concern about the legitimacy of the system”.

On fairness, several respondents raised the issue of unconscious bias. One reported “a lot of unconscious bias towards African, Asian and Commonwealth of Independent States people”, including a tendency to distrust their witness evidence more readily. Another observed that “white tribunals have demonstrated prejudices and bias against Latin American parties and lawyers”. These reflect the lived experience of practitioners in the system, and the data suggest they’re not isolated views or incidents.

Opening submission

This report represents the first empirical baseline for a conversation that has so far largely proceeded without one. In that spirit, it raises more questions than it resolves. Should tribunals reflect the ethnic backgrounds of the parties, or aim for broader systemic diversity? Does the answer change depending on whether the dispute is commercial or investor-state? What role do closely connected factors such as legal tradition and language proficiency play, and how do we disentangle them from ethnicity itself?

The data don’t provide neat answers. What they do provide is evidence that the international arbitration community, by a large majority, considers ethnic diversity on tribunals to matter – for legitimacy, for fairness and for the quality of decision-making. That is worth knowing. The real-world examples exposed by the research suggest that the consequences of getting this wrong are concrete: a witness misjudged, a cultural cue missed.

The full report is available on the Cortex website, as well as on the IBA’s website, where it is accompanied by the IBA Arbitration Committee’s own commentary on the project.

Keynote: Canada Arbitration Week 2025, Montreal 

Let me start with two questions: 

1. How do tribunals actually think? 
2. Can we redesign Tribunal deliberations to promote better decisions? 

A lot has been written about cognitive bias in international arbitration – covering things like confirmation bias, recency bias or – the worst offender – overconfidence.

Cognitive glitches like these are critical to be aware of if we want to improve our own thinking. But the conversation usually stops there – at the level of the individual. This assumes that each arbitrator is a discrete and self-contained decision-maker.

But most arbitral awards are not produced by individuals. They’re produced by groups of individuals – usually panels of three. And that changes everything.

So today I want to offer a different lens: to look at decision-making in international arbitration not at the level of individual psychology, but the more complex, the dynamic level of tribunal cognition: how tribunals think, feel, influence and decide as a group. 

Pitfalls of Group Decision-Making 

According to a recent study, 78% of people want to hear bad news first – so here it is: 

Group decision-making might sound fair, democratic and rigorous. But psychology tells a more complicated story. There are two particular hazards I want to highlight here.

1. Groupthink 

The first is groupthink. This is a classic psychological phenomenon where a group’s desire for harmony leads it to make consensus decisions that are made without thorough or complete critical evaluation – and often result in irrational or flawed outcomes.

According to Irving Janis, the Yale psychologist who pioneered the research on this topic, the main factor contributing to groupthink is the degree of cohesiveness, amiability, or what he called esprit du corps – within the group. This is the feature that leads individuals to prioritise maintaining friendly relationships and to avoid conflict over raising controversial ideas or independent thinking.

There are several side-effects of groupthink. Most relevant in this context, we have: 

• The illusion of invulnerability. This gives a tribunal an inflated sense of certainty that their decision is the right one. Overconfidence at a group level in other words. 

• Self-censorship can occur, where individuals withhold from expressing dissenting views or from questioning the current decision. 

• And then the illusion of unanimity, where silence from individuals is – wrongly – perceived to mean that they agree. 

These symptoms can have fundamental, sometimes disastrous consequences.  

A classic real-world example is the Space Shuttle Challenger disasterin January 1986. There was intense media interest in this particular launch because it was the first mission involving a civilian crew member – school teacher Christa McAuliffe. So NASA project leaders were under even greater pressure than usual – in particular not to delay the launch. 

But weather forecasts were predicting record low air temperatures for a shuttle launch – minus 8 Celsius overnight and minus 3 at launch time. This was a problem. The engineers who worked on the solid rocket boosters were concerned that the cold air would impact the resilience of the booster’s O-rings. These are thick rubber rings 38 feet in circumference that sit between the joints of the rocket, sealing them together. 

The evening before the launch, the engineers voiced their concerns to the NASA project team, recommending delaying the launch until temperatures were over 12 degrees Celsius – where the O-ring’s performance had actually been tested. But with the backdrop of media pressure, NASA pushed back heavily against the engineers’ concerns and the engineering firm’s leadership eventually relented – giving a green light. 

73 seconds after blast-off, the O-ring seal failed in the right solid rocket booster causing hot pressurised gas to leak into the joint and burn into the external propellant tank – as a result, the shuttle stack was propelled with forces that tore the orbiter apart and all 7 crew members died. 

Groupthink here meant that NASA’s strong desire to obtain consensus and to avoid delay overrode critical safety concerns, leading to a catastrophic decision without proper evaluation of the risks. 

There are several potential mechanisms behind groupthink: 

• Deferring to others’ preferences in order to preserve harmony or social capital.
• Failing to share knowledge.
• Underweighting dissenting voices.
• Social loafing – when individuals are less likely to think for themselves, because they are in a group.
• Amplifying individual cognitive bias, like those we mentioned at the start.

Given the culture of international arbitration, the strong social network element, the camaraderie – and the pressure arbitrators may feel in preserving future appointment opportunities, groupthink is one of the most significant risks to be aware of in tribunal decision-making.  

2. Group polarisation 

The second decision hazard in tribunal psychology is group polarisation. This refers to the tendency for group discussions to lead members to adopt more extreme positions than they originally held as individuals.

Interestingly, polarisation is a much greater risk where groups are newly-formed – and where they are undertaking new tasks such as making a novel decision. So highly relevant to arbitral tribunals. 

Unfortunately, group polarisation has been found alive and well in many legal contexts. For example, jury studies show that after deliberating together, mock jury members often decided on punitive damages awards that were either larger or smaller than the amount any individual juror had favoured prior to deliberation. When jurors favour relatively low awards, discussion would lead to an even more lenient result; by contrast, if the jury was inclined to impose a stiff penalty, discussion would make it even harsher. 

Another classic study, conducted in 1973, looked at whether Federal District Court judges in the US made different decisions when sitting alone versus in small groups.  Judges were given a hypothetical case involving a challenge to the constitutionality of a statute or government policy, and the judges had to choose between a cautious course of action – upholding the statute or policy – or a more extreme one – striking it down as unconstitutional. 

When sitting alone, judges opted for the extreme action about 35% of the time. But when sitting in panels of three, that figure jumped dramatically to 65%. 

So we know that even highly trained legal minds are not immune to group dynamics.

In fact, group settings can significantly increase the likelihood of incomplete or extreme decisions.

There are a few mechanisms at play here but primarily, group polarisation happens because individuals become more convinced of their own views when they hear new and persuasive arguments supporting them. Relevantly for tribunal cognition, this sort of ‘informational influence’ is especially strong in discussions involving complex, intellectual issues.

So that’s the bad news for tribunal cognition. 

Cognitive Assets of the Tribunal

Now the bit you’ve all been waiting for: the good news.

Collective intelligence 

Research shows that – under the right conditions – groups can be smarter than any of their members. This is similar to the wisdom of crowds effect – and is the promise of the relatively new science of collective intelligence.

The seminal study here was published in 2010 by a group of MIT researchers lead by Anita Woolley.  They wanted to know whether it was possible to measure the intelligence of a team or group – in the same way that we measure general intelligence or IQ of individuals. 

They took 192 groups of people and gave them various problem-solving tasks like brainstorming, making collective moral judgments, negotiating. 

Using factor analysis, they found that each group had an identifiable, measurable “c factor” which explained its test scores and could accurately predict its performance on future tasks. This is collective intelligence or CQ. 

Naturally, they wanted to know what lead to higher CQ. And interestingly, variables you would have thought played a role – didn’t: 

• There was no correlation between a group’s CQ and motivation, satisfaction or team cohesion. 

• There was also only a weak correlation between the group CQ and the IQ of the individual members. What this means is that a group of highly intelligent individuals can perform horribly as a team, while a group of average IQs may outstrip the Mensa members on collective tasks.

There was another surprising finding in this research I want to share. And that is – the more women in the group, the more intelligently it performed – though there was an interesting drop-off for all-women groups.

Now in case the cognitive drawbridge has started to raise among approx. 50% of you, some important caveats: 

• First, as I said, the effect fell off with all-women teams. So groups need gender mix to function at their best. 

• More fundamentally, it isn’t gender per se that mediates the CQ boost. Further analyses were done on all of the groups and what the researchers found is that the key variables predicting intelligence were turn-taking within conversations and, relatedly, social sensitivity. 

• And in general, women score more highly than men on measures of social sensitivity – hence the unexpected gender result in the study. 

The research on collective intelligence shows that it’s not just who’s in the tribunal – but how they interact – that affects the quality of its functioning. 

Cognitive diversity 

This brings us to cognitive diversity. Cognitive diversity reflects differences between individuals in knowledge, processes or actions within a group context. This is distinct from – though may stem from – demographic differences such as gender and ethnicity. 

In general, studies in this area find that when groups include diverse thinking styles, they perform better – especially on complex tasks.

One study looked at how groups approached solving a classic logic puzzle – the Wason Card Sorting Test. For your own entertainment, this is the test: 

• You have 4 cards in front of you: 2 letters, 2 numbers. 
• You have to answer this question: Which card(s) should you turn to test the rule: “All cards with vowels on one side, have an even number on the other”.
• The intuitive but wrong answer is to turn over 1 even number + 1 vowel.
• The correct answer is to turn over 1 odd number + 1 vowel.

Researchers took data from 500 small groups completing this puzzle, and analysed the transcripts of the group conversations and compared the groups’ initial answers with their answers following the joint discussion. What they found is that greater cognitive diversity – measured by group size, diversity of initial ideas expressed and diversity in discussion content – consistently led to more successful group deliberation and better decision-making.

There is important nuance here. Too much diversity can actually hamper group function – particularly where it facilitates intragroup conflict or hinders agreement on common goals. 

One study looking at different cognitive styles – that is, the way that team members encode, organise and process information. They found a curvilinear relationship – an inverted U-shape – between the degree of cognitive diversity in the group and its collective intelligence.

So, there may be a Goldilocks degree of difference. But difference is essential for optimal group intelligence and decision-making. 

Neural synchrony 

Here’s where neuroscience adds something exciting. Hyperscanning is a relatively new method where researchers scan multiple brains at once – brains that are actively interacting with one another in real time.

Usually, individual brains operate in their own unique way, with electrophysiological brain waves – the rhythmic electrical patterns produced by neurons in the brain – appearing independent when compared side by side. But recent hyperscanning studies show that when people cooperate effectively, their brains literally start to sync up. Brain waves fall in line such that neural synchrony increases in well-functioning groups. This synchrony correlates with – and can reliably predict – better collaboration, better decisions and even greater trust.

In other words: thinking together is not just a metaphor. It’s biological.

So the cognitive assets of a tribunal, in intelligence, in diversity, are important considerations with real neural implications for the way they will behave as a group. 

How Tribunals Actually Function

So what actually happens inside a tribunal?

Let’s talk briefly about some of the invisible dynamics that shape group deliberation. These are little talked about because they are hard to pin down – but they are also the most powerful forces that shape interpersonal dynamics like those within international tribunals. 

Empathy & Trust 

First, empathy and trust are the bedrock of constructive group work – but they are fragile, especially in culturally diverse panels.

Neuroscientific studies show that when people feel understood by others in a group, their brains become active in reward-related areas – those linked to food, money, recreational drugs. But when people feel excluded or misunderstood, you see brain activity in areas linked to social pain – fascinatingly, the same brain areas that code for physical pain.

Now think about tribunals in international arbitration. You may have different cultural backgrounds, you have intense time pressure, you may have language barriers, different legal traditions. That’s a tough environment for empathy and trust – and something which may need sensitive, deliberate work to develop. 

Influence 

Then we have dynamics of personal influence. 

Many factors influence our influence. Status, seniority – so are you chairperson or wing?  Confidence is important – it’s not what you say but how you say it. Again, there may be important cultural variations here in expressed or perceived confidence. 

Something as apparently benign as who speaks first can make a huge impact to influence within a group. Jury studies which analyse patterns of jury deliberation find that in 90% of cases the preferred verdict at the start of deliberations ultimately became the jury’s final verdict. So think carefully about who is speaking, when – and what we take from that communication. 

Ingroup bias 

Finally, we have ingroup bias. This is where we unconsciously favour ideas from people we identify with – and ignore or critique ideas from those we don’t. 

Identity here can be very broad – it could be something obvious like gender, age or nationality. But it might be something like shared background, native language, law firm – or even something as trivial (I should not say trivial!) as liking the same hockey team.

These invisible dynamics show us that tribunal deliberations aren’t just a clean debate of legal arguments between rational actors. They are a complex dance of emotion, identity and influence.

What This Means for Arbitration 

So what do we do with all this?

Luckily, armed with relevant insights from psychological science, we can design tribunal culture to improve cognition and decision-making within the group. 

Practical strategies should start at the beginning – with appointments. 

1. We should look beyond identity and appoint for cognitive diversity – knowing that difference unlocks optimal performance. Yes, gender, nationality and legal tradition matter. But so do experience diversity, thinking styles and even temperament. Is this someone who asks questions or makes statements? Someone who changes their mind, or rarely does?
2. Tribunals can take an organised and structured approach to deliberations. Encourage turn-taking. Ensure that you’ve heard – and really heard – from everyone. Communication – not just of opposing ideas but of knowledge others might have missed – is repeatedly found to be critical in enabling a group to reach the best decision. Communication is – as ever – key. 
3. Relatedly – the Chair should avoid giving their view first. Chairs are often more senior, more experienced, as a result (perhaps?) more confident. But the natural status of the role will also serve to anchor all subsequent discussion and consideration. Newer arbitrators in particular may feel uncomfortable disagreeing with a senior, experienced arbitrator, regardless of how preliminarily their view is expressed.
4. Individual arbitrators can be assigned the specific role of devil’s advocate. Perhaps different devils for different issues. The best devil’s advocacy is where it arises naturally – because people actually disagree – but in case all tribunal members appear to agree at least initially, this formal role can ensure that ensure prevailing views are adequately stress tested.
5. Another idea is to invite written reflections from each member of the tribunal before oral deliberations begin. This can help ensure that individual perspectives are fully formed and independently considered, reducing the risk of groupthink, premature consensus or dominant voices unduly shaping the discussion.

These procedural designs are not gimmicks. They are methods of scaffolding better cognition. Of helping a tribunal think more effectively as a collective.  

We should think of tribunals as cognitive ecosystems. Systems that generate their own reasoning. And systems that can be tuned and supported.

One Good Idea 

And so, in the spirit of this conference’s theme – One Good Idea – here’s mine:

Consider Tribunal Intelligence. This idea that each Tribunal will demonstrate different degrees of collective intelligence depending on the various factors we’ve talked about today. 

With that in mind, if we want better decisions, we can’t just focus on evidence and procedure. We need to support the way tribunals think – as a collective brain.

The challenge – and the opportunity – is to engage in conscious deliberation design to promote tribunal intelligence and better decisions.

Et voilà.

Dr Ula Cartwright-Finch 

Montreal, June 2025

Keynote: Edinburgh International Arbitration Festival 2024

One of the most exciting parts of my studies at St Andrews was discovering how the brain can change. 

Neuroplasticity 

50 years ago, the idea that brain anatomy could change at all was unthinkable. It was widely accepted that while our brains are malleable when we’re young, they gradually harden into a fixed structure by the time we’re adults – like clay in a kiln. 

It was also believed that we are born with a fixed number of neurons, and once they’re lost through damage or decay, they can never be replaced. Hence, “you can’t teach an old dog new tricks”.

Thanks to modern neuroscience, we now know that the adult brain changes continuously throughout our lifetime – in response to what we experience and what we do. When we practice a skill, or repeat a thought, connections between neurons get stronger and specific brain regions get bigger through neurogenesis – the growth of new brain cells – or by the formation of new pathways. 

The net effect of this neuroplasticity is that we get better at the activities and thoughts we rehearse. So instead of hardening clay, the brain is more like playdough. 

Let me give you an example.

In the Burmese archipelago, and off the West Coast of Thailand, nomadic tribes called Sea Gypsies spend most of their lives at sea. They swim before they can walk, and they live on clams and sea cucumbers, which they have to dive very deep to find. One tribe can dive over 75 feet without equipment – which, for reference, is about 12 Sean Connerys. What makes Sea Gypsies unique for our purposes is that they can see clearly at these great depths – without goggles. 

Most of us can’t do this because when sunlight is refracted underwater, our visual system can’t interpret the light that hits our retinae. But the Sea Gypsies have learned to control the size of their pupils – shrinking them by 22% in order to see underwater. This is remarkable. We always thought that the way our pupils adjust to the light – getting larger in the dark and underwater – was an unconscious and automatic reflex. 

In fact, the Swedish researcher who studied the Sea Gypsies taught this underwater ability to Swedish children – proving that what was once thought to be a reflex is in fact something our brain can learn to control. 

International arbitration culture

This adaptability highlights how our brains change as a result of the culture we grow up in. International arbitration has its own distinctive culture. There are signature activities we learn and we follow: 

• the directions we capture in procedural orders; 
• the system we use to label exhibits;
• the way we address witnesses; 
• even our diet during hearings – mostly biscuit-heavy. 

But are these signature activities truly optimised to suit the way our brains work?

In this talk, we’re going to look at instances of mis-match between the way our brains function and certain elements of international arbitration culture. We’ll do this by exploring 3 features of human psychology linking them to arbitration.  

• At the individual level, the limits of human cognition.
• At the interpersonal level, our inherent sociability.
• At the environmental level, the impact of external conditions on our thinking.

1. Our brains have limits. 

First: while it isn’t entirely comfortable to admit, our brains have limits. Millions of pieces of information hit our sensory systems every second – but we can only process and understand so much. 

Let’s look at some basic numbers. 

Memory and attention 

The capacity of our short-term memory – that is, the ability to hold new information in our head until it becomes long-term knowledge – is about 7 bits of information, give or take 2. 

Our ability to focus attention? Also not great. The average attention span is anywhere between 2 seconds to around 48 minutes. 

We can improve our memory and our attention a bit – for example, with certain types of meditation – but they both get worse when we’re stressed or sleep-deprived. Not great news for most of us in this room. 

But what actually happens when we reach our cognitive capacity? Our decision-making, our memory – even our ability to detect information at all – begins to falter. 

Recency bias

For example, when we are given lots of information over a period of time – like a tribunal over the course of a proceeding – we remember and we are influenced by the most recent bits much more than those sandwiched in the middle. 

This is Recency Bias. One reason why Counsel (and arbitrators) love a post-hearing brief.

Vividness effect

Next, the vividness effect. Compared to bland or abstract facts, information that is emotionally striking, rich in sensory detail, is more likely to capture our attention, be remembered and influence our decisions. 

This explains why I can still remember where I was the first time I heard an advocate comparing shares in a company to a dead cat.

Confirmation bias 

Then, confirmation bias, where we selectively search for evidence that supports our view and we ignore, forget, explain away or otherwise downgrade evidence that goes against it.

According to a large survey in 2012, 88% of arbitrators – as a matter of practice – or on at least one previous occasion – form a preliminary view of the case by the time they walk into the merits hearing.

In view of the research on confirmation bias – how our perception, evaluation and recollection of evidence changes once we form an initial hypothesis – this relatively common practice deserves careful consideration. 

Overconfidence

And finally, overconfidence.

In an interview in 2015, the late, great Daniel Kahneman said, if he had a magic wand, this was the one bias he would eradicate. Ironically, few people appreciate that there are actually 3 distinct types of overconfidence.

• Overestimation: believing you are better than you actually are at a task or a skill. Like planning 30 minutes for something that takes you 3 hours.

• Overplacement: thinking you’re better than others. 

• Overprecision: excessive certainty that your predictions, your calculations of risk, are accurate. 

This last variant of overconfidence is especially dangerous – because it stops us questioning our assumptions, challenging our initial conclusions, and remaining open to new evidence – ultimately undermining the fairness and accuracy of our decisions.

Why might we need to worry about this in arbitration? 

Think about your long-suffering, document-deluged arbitrator. By the time they reach the hearing, they are supposed to have read, remembered, and understood thousands of pages of submissions, evidence, expert reports – not even counting the exhibits. They might be jet-lagged. They might be stressed. Their memory is almost certainly full trying to remember a new cast of characters. 

If you were to design the perfect environment for cognitive overload, this would be it: an environment where the sheer volume of information forces us to rely on mental shortcuts to keep up. 

So, my first point – in case you have forgotten – is that our brains have a bandwidth limit. When we exceed our data limits – as we so often can – our brains take shortcuts, and these shortcuts can create flaws in our decisions. 

2. We are ‘group-y’

We are inherently ‘group-y’. One of the most distinctive and significant traits our species has evolved is the tendency – automatically and immediately – to form groups with other people.

The 1968 cult classic, Planet of the Apes, offers a fascinating case study of this instinctive habit. In the film, there are 4 species of ape: humans, chimps, gorillas and orangutans. On the first day of filming, actors donned their designated ape costumes. Later that day, during lunch in the cafeteria, the director noticed something strange. Despite being strangers and still in their costumes, the actors instinctively grouped together at tables according to their ape species. Gorillas sat with gorillas, chimps sat with chimps – perfect segregation. 

What about arbitration? 

If I didn’t know about overprecision, I’d put money on international arbitration being one the group-iest branches of law in the world. So far so group-y. So what? 

In-group bias 

Well, after we have mentally divided the world into different groups – those to which we belong, and ‘the others’ – we think, behave and feel differently towards them. 

One important factor is that we like people like us. We prefer to be around people we share some important personal characteristic with, where we share an affinity. Often this is some visually obvious characteristic – gender, age, nationality – but it could be anything: same law firm, shared love of a particular sport, just the mutual belief that deep-frying a Mars bar is a perfectly acceptable dinner choice.

If we reflect on this, it’s something we experience all the time. At events like this, it feels easier – we are more relaxed – speaking to people we feel similar to. Conversely, it feels a bit harder – and we shy away – from speaking to people we feel different from.

The Usual Suspects Effect 

This impacts many moments in an individual’s career. One specific instance is the selection of arbitrators.

Last year I published the results of a research project designed to map systematically for the first time, how parties – or more accurately, how Counsel – go about choosing an arbitrator. Spoiler: there is no system. This critical decision-making process is largely ad hoc. 

But there were some commonalities. One was that, almost universally, the first method for coming up with potential candidates was what I call the ‘mental rolodex approach’ – where you sit back in the chair and think of names that pop into your head. 

What’s wrong with this approach? Well, if you take affinity bias and add recency bias, what happens is Counsel coming up with the names of people they spoke to at last week’s conference, or who they saw and liked on a recent panel.

Given that arbitration lawyers are themselves from a relatively un-diverse range of backgrounds, this inevitably leads to a relatively un-diverse pool of arbitrators – hence, the Usual Suspects effect. 

To wrap up my second point: we have natural instincts that shape how we connect with others. These instincts have been wired into us for survival – so they are very powerful. But they may not help us create the best outcomes in arbitration.

Breaking free from them requires conscious, deliberate effort – good intentions aren’t enough. So, my challenge for you today is simple: notice who you’re naturally drawn to – and then, make the effort to connect with someone else, as well.

3. The impact of external conditions 

Third, and finally, our brains operate very differently under different conditions.

Deadlines and caffeine 

Many of us have no doubt experienced firsthand that nothing motivates quite like a deadline. Counterintuitively, research shows that the more time we have, the less likely we are to do something. So, keep that in mind the next time you’re asking someone to complete a task

Caffeine works wonders on our energy and our focus because it blocks adenosine receptors – the neurochemical that makes us feel sleepy. On the topic of coffee – one image I cannot forget is from a NASA study looking at the effects of various substances on spiders’ ability to weave their webs… 

The foreign language effect

One condition of particular interest in international arbitration is language. 

Nelson Mandela said “if you talk to a man in a language he understands, that goes to his head. If you talk to him in his language, that goes to his heart”.  

Science may think this an exaggeration but he had a point. What is clear is that we feel emotions much more strongly in our mother tongue than when we hear or read the same thing in a foreign language. This is most likely because there is a much stronger social context surrounding our learning of our native language. 

One study from the fascinating field of neurolinguistics, asked German participants who could also speak English as a foreign language to read exciting passages from Harry Potter in both languages. Brain imaging showed that the amygdala – the part of the brain that processes emotions – became activated only when participants read the exciting passages in German but not in English. 

As a result of this difference in emotional processing, language also influences our judgments and our decisions. When people assess something risky in their mother tongue, they over-rate the benefits and under-quantify the risks. 

We also respond differently to moral decisions. You may be familiar with the trolley problem: 

‘A train is very quickly approaching 5 people. It has a problem with its brakes and cannot stop. Those 5 people will die if the train continues on its path. There is a way for you to divert the train, but that would result in a man dying.’ 

In a famous study by a team in Barcelona, for those people who read this rather unpleasant dilemma in their first language (Castilian Spanish or Catalan), only 17% sacrifice the 1 man for the 5. But when they face the same dilemma in their second language (English), that figure increased to 40%.

This finding has been replicated several times in several languages – so it isn’t the case that English makes us coldly rational. 

In international arbitration, where judgments and decisions are made by many people across multiple languages, this is a fascinating factor that cannot be overlooked. 

So point 3 is that different external conditions change the way we think and behave – in arbitration as in life. 

Conclusion

We share 98% of our DNA – our genetic code – with chimpanzees. And thanks to the Human Genome Project, we know exactly which genes differ between us. One of them is the gene that controls neuron production. Neurons are created through cell division – 1 becomes 2, 2 becomes 4 etc – and this continues longer in us than in chimps, so we end up with about 100 billion neurons; whereas chimps have brains a third the size. 

Because each single brain cell can be connected to thousands of other cells, we have an exponentially greater number of neural connections. Think – 10 followed by at least a million zeros. This is why the brain is often described as the most complex object in the known universe.

It’s also why we are capable of massive and profound, structural change. So it’s time to learn some new tricks. To think about the typical systems and processes we use in arbitration – and ask ourselves, do they truly support the way we think and make decisions?

With intelligent adaptations, guided by psychological science and research, we can shape an international arbitration culture that is designed to work with our brains, rather than against them. And that is my case for change.  

Dr Ula Cartwright-Finch

Edinburgh, September 2024