Doing good statistics

The organisation “Inside Justice” – has shown interest in Ben’s case and has published an essay by Statistician Richard Gill, text here below.
The original can be found on the INSIDE JUSTICE website.

“Imagine if you asked people to roll eight dice to see if they can ‘hit the jackpot’ by rolling 8 out of 8 sixes. The chances are less than 1 in 1.5 million. So if you saw somebody in the UK – let’s call him Fred – who has a history of ‘trouble with authority’ getting a jackpot then you might be convinced that Fred is somehow cheating or the dice are loaded. It would be easy to make a convincing case against Fred just on the basis of the unlikeliness of him getting the jackpot by chance and his problematic history.

“But now imagine Fred was just one of the 60 million people in the UK who all had a go at rolling the dice. It would actually be extremely unlikely if less than 25 of them hit the jackpot with fair dice (and without cheating) – the expected number is about 35. In any set of 25 people it is also extremely unlikely that there will not be at least one person who has a history of ‘trouble with authority’. In fact you are likely to find something worse in the UK, since about 10 million people there have criminal convictions, meaning that in a random set of 25 people there are likely to be about 5 with some criminal conviction. And plenty of others will have other dark events in their life histories.

“So the fact that you find a character like Fred rolling 8 out of 8 sixes purely by chance is actually almost inevitable. There is nothing to see here and nothing to investigate. Many events which people think of as ‘almost impossibe’/’unbelievable’ are in fact routine and inevitable: see for instance this article on “Impossible Events” or Norman Fenton and Martin Neil’s book “Risk assessment and decision analysis with Bayesian networks ” section 4.6.3 “When Incredible Events Are Really Mundane”, or the book “The Improbability Principle” by David Hand (past president of the Royal Statistical Society).

“Now, instead of thinking about ‘clusters’ of sixes rolled from dice, think about clusters of patient deaths in hospitals. Just as Fred got his cluster of sixes, if you look hard enough it is inevitable you will find some nurses associated with abnormally high numbers of patient deaths. In Holland a nurse called Lucia de Berk was wrongly convicted of multiple murders as a result of initially reading too much into such statistics (and then getting the relevant probability calculations wrong also). There have been other similar cases. In the UK it seems that Ben Geen may also have been the victim of such misunderstandings.”

The dice throw analogy is very accurate, when we look at “health care serial killers”. A typical full time nurse works roughly half of the days of the whole year (take account of holidays, training courses, absence due to illness, “weekends”), and then just one of the three hospital shifts on a day on which she works. That makes 1 in 6 shifts. So if something odd happens, there is 1 in 6 chance it happens on his/her shifts. (However … more incidents happen in weekends and some nurses have more than average weekend shifts). But there is the question: which shift did some event actually happen in? There’s a lot of leeway in attributing some developing medical crisis situation to one particular shift. Then there is the question, which shift did the nurse have? There is overlap between shifts, and anyway, sometimes a nurse arrives earlier or leaves later. This gives hospital authorities a great deal of flexibility in compiling statistics of shifts with incidents, and shifts with a suspicious nurse. Both in the Ben Geen and in the Lucia de Berk case, a great deal of use was made of this “flexibility” in order to inflate what seems to have been chance fluctuations into such powerful numbers that a statistical analysis becomes superfluous: anyone can see “this can’t be chance”. Indeed. It was not chance. The statistics were retrospectively fabricated using a prior conviction on the part of investigators (medical doctors at the same hospital, not police investigators) that they have found a serial killer. After that, no-one doubts them.

What happened in both Lucia and Ben’s case is therefore not only the surprising coincidence but the magnification of a coincidence after it has been observed. Doctors look back at past cases and start reclassifying them. So the dice analogy is not quite correct: it is more like you see someone rolling 5 out of 8 sixes (and it’s someone you think is a bit odd in some way), and then you turn over the three non-sixes and make them into sixes too. Then you go to the police: “8 out of 8”. This is exactly Lucia: 9 out of 9 – but actually three or four of those dice outcomes had been altered. It’s just the same for Ben Geen. He was even convicted of causing the respiratory arrest of a patient who was admitted to the emergency ward on which Ben worked … after all there were so many of those events and always it seemed when he was on duty … yet it turns out that that particular arrest had already occured in the ambulance on the way to the hospital! It seems nobody checked.

Returning to the 8 dice throw analogy, a further twist to the story is that you never took the trouble to look at a further 20 dice-throws which had also been done and in which, surprise surprise, there are only two or three sixes. The eight dice throws which came to your attention are ones which you remember. Back to the hospital, you remember those 8 events precisely because that striking nurse about whom people have been gossiping was there. Neither in the Lucia case, nor in the Ben Geen case, did anyone take the trouble to go through all the shifts when Lucia or Ben had not been present, investigating the records to see if incidents had actually happened there too: incidents which should have gone into the statistics.

Doing good statistics might be not much more than common sense, it might not be rocket-science … but it is something which is incredibly easy to get very very wrong, with devastating effects.

The Norman Fenton blog



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s