Faculty of Social Sciences
Inaugural Lecture Series 2024/25

Estimating Ethnic Disparities in Sentencing:
Methodological Challenges and a Path Forward

Jose Pina-Sánchez

Thanks Louise for such nice introduction, I am not good at taking complimnets so I am beyond embarrased right now.

And thanks Paul and Izzy for organising this series of Inaugural lectures, which give us the opportunity to come together and learn from the great research taking place in our faculty.

Ok, this is a talk about methods, so I apologise in advance. When Izzy approach me to see if I wanted to give one of these inaugural lectures she highlighted that I should avoid using technical jargon, and I tried, like for a few hours, but I found that I could not so I gave up, so you will have to bear with me.

The reason for that is because the main point I want to make is that research methods matters, how we should highlight their importance, invest in their teaching and development, and how without them we are in many ways fooling ourselves.

But to illustrate that underlying message I am going to focus on the topic of ethnic disparities in sentencing. A topic that is quite important both legally and politically, and consequently central to much of the research we undertake in the faculty, in particular the disparities/discrimination angle, while the sentencing part also makes it central to some of the research we do in Law and criminology. So I am hoping that it won’t be too boring.

Background

Sentencing

The strongest expression of the coercive powers of the state
Clear material consequences
Equally important symbolic value

Disparities in England & Wales

The Lammy Review (2017)

– 240% higher odds of custody for minority drug offenders

– side note: 2.4 odds ratio \(=\) 140% higher odds
Is this evidence of discrimination?
No definitive answer

– how to interpret the evidence seems a personal choice

– we have not done our job

The question, of course, is, is it really blind?

David Lammy, the current foreign secretary - a politician I really like by the way, as he is one of the few that really stood up against the Brexit rethoric in all these years - he was commissioned by David Cameron to author an independent review of ethnic disparities in the criminal justice system in England & Wales.

In the chapter on sentencing, the Lammy Review highlighted a data point that was quite revealing. 240% higher odds of custody for ethnic minority offenders who are sentenced for a drug offence, compared to their white counterparts.

This data point was derived from an older MoJ report. They used a representative sample of offenders sentenced in the Crown Court, and controlled for guilty plea, previous convictions, and age and sex of the offender.

Side note: the true figure is 140%. There you go, even an independent review commissioned by the UK government can also make massive gaffes, which btw continue to be propagated and uncorrected. I will get back to that later.

Is this evidence of discrimination? What do you think? Raise your hands if you think that is the case. Ok, anyone who might consider that this is insufficient evidence to claim there is discrimination in sentencing in England and Wales? And why would you think that?

That is the position adopted by the Sentencing Council, the Ministry of Justice, and most of researchers working on this area, including myself back in the day, most of us caveat our findings indicating that the presence of such disparities does not equate judicial discrimination.

That is because this is a harder question that it might appear at first sight. When we make these kind of blunt comparisons with real world (non-experimental) data, it is likely that they are biased. It is possible that the two groups are not entirely comparable, and that the observed disparities are legitimate. For example, we know that black offenders are less likely to plead guilty, and do so at a later stage, regarding drug offenders specifically, it also seems to be the case that the quantities and types of drugs with which white offenders are charged are lower and less dangerous.

No definitive answer. How to interpret this evidence seems a personal choice. I am going to argue that this is suboptimal position to be in, and the result that we have not looked into the questions in enough detail. There are questions for which the data is not good enough to have a clear answer, but there are also instances where going in more depth, using better methods, and just plainly being more throrough, we can come up with definitive answers. And I am going to try to make the case that this is one of them.

Real-world implications

A. There is discrimination but we fail to acknowledge it

– we perpetuate a terrible injustice
B. There is no discrimination but we claim there is

– we undermine trust in the criminal justice system

– alienate minority groups

– waste efforts by focusing on the wrong crisis

Methodological challenges

Confounding bias

Practically impossible to make ‘like-with-like’ comparisons

Post-treatment bias

We might be inadvertedly controlling away judicial prejudice

– e.g. offender’s remorse and rehabilitation potential

Misclassification & missing data

White offenders are not a homogeneous group
Different patterns of missing data depending on who identifies offender’s ethnicity

Framing bias

‘Race studies’ show twice stronger disparities than other sentencing studies
Systematic cherry picking

– every study in E&W highlights the “240% higher odds”

– we only see disparities in 1 out of 9 offence groups

So, can we tell whether there is discrimination?

We can estimate some uncertainty

We can evaluate bias qualitatively

Sentencing Council (2021) conditioned on all factors listed in their guidelines

– still found significant disparities (40% higher odds)
Missing legal factors isn’t a sufficient condition for statistical bias
No missing legal factor is realistically strong enough to render those disparities non-significant

Anticipate the direction of the bias

Disparities could be understimated

– the Council controlled for mitigating factors we suspect are ‘racially-constructed’

– did not differentiate between groups of whites
We do not find evidence of bias in missing data

My conclusion

We can provide a definitive answer

– there is sentencing discrimination in England & Wales

– not as widespread as commonly believed
We should redouble our efforts in tackling the problem

– reminders in guidelines, pre-sentence reports, legal aid

An in-depth analysis of the underlying assumptions and bias affecting a given question allow us, in this case, to provide a definitive answer.

I would posit that this precision is highly useful, rather than providing a generic statement, delve into the question and locate exactly where the problem is. Such anlysis is far more useful than 10 or 20 another superficial studies of the kind that we are accostumed to publish, and I certainly include myself here.

Sentencing is one of the simplest subjects we can think of in the social sciences. Clearly confined in time and space, well identified agents, clearly defined outcomes, it is a joke, compared to what most of you do throughout the social sciences.

We need a radical change in how we do research. We need to redouble our efforts to minimise bias. We need to get better at assessing the robustness of the evidence. So we can avoid overconfidence but also undue underconfidence. Studies that are confidently giving us a finding that is objectively incorrect, or that they are unnecessarily caveating their findings so much that anyone can make of the results what they want.

How Do We Move Forward?

Invest in research methods

Every researcher should be able to identify research assumptions

– and assess how they could lead to bias
Empirical researchers should be able to estimate those biases

– and reflect them in the uncertainty of their findings

A Messy Trajectory

Change of research culture

Slow research

– quality over quantity
Reconsider measures of prestige

– emphasis on citations lead to bias

– incentivise replications
Open research

– registered reports

If we try hard enough, and do not fool ourselves along the way, we can answer pretty hard questions