Bad Science

where there are three bullet holes close to each other, then draw a target around them, announcing proudly that I am an excellent marksman.

You would, I think, disagree with both my methods and my conclusions for that deduction. But this is exactly what has happened in Lucia’s case: the prosecutors found seven deaths, on one nurse’s shifts, in one hospital, in one city, in one country, in the world, and then drew a target around them.

This breaks a cardinal rule of any research involving statistics: you cannot find your hypothesis in your results. Before you go to your data with your statistical tool, you have to have a specific hypothesis to test. If your hypothesis comes from analysing the data, then there is no sense in analysing the same data again to confirm it.

This is a rather complex, philosophical, mathematical form of circularity: but there were also very concrete forms of circular reasoning in the case. To collect more data, the investigators went back to the wards to see if they could find more suspicious deaths. But all the people who were asked to remember ‘suspicious incidents’ knew that they were being asked because Lucia might be a serial killer. There was a high risk that ‘an incident was suspicious’ became synonymous with ‘Lucia was present’. Some sudden deaths when Lucia was not present would not be listed in the calculations, by definition: they are in no way suspicious, because Lucia was not present.

It gets worse. ‘We were asked to make a list of incidents that happened during or shortly after Lucia’s shifts,’ said one hospital employee. In this manner more patterns were unearthed, and so it became even more likely that investigators would find more suspicious deaths on Lucia’s shifts. Meanwhile, Lucia waited in prison for her trial.

This is the stuff of nightmares.

At the same time, a huge amount of corollary statistical information was almost completely ignored. In the three years before Lucia worked on the ward in question, there were seven deaths. In the three years that she did work on the ward, there were six deaths. Here’s a thought: it seems odd that the death rate should go down on a ward at the precise moment that a serial killer – on a killing spree – arrives. If Lucia killed them all, then there must have been no natural deaths on that ward at all in the whole of the three years that she worked there.

Ah, but on the other hand, as the prosecution revealed at her trial, Lucia did like tarot. And she does sound a bit weird in her private diary, excerpts from which were read out. So she might have done it anyway.

But the strangest thing of all is this. In generating his obligatory, spurious, Meadowesque figure – which this time was ‘one in 342 million’ – the prosecution’s statistician made a simple, rudimentary mathematical error. He combined individual statistical tests by multiplying p-values, the mathematical description of chance, or statistical significance. This bit’s for the hardcore science nerds, and will be edited out by the publisher, but I intend to write it anyway: you do not just multiply p-values together, you weave them with a clever tool, like maybe ‘Fisher’s method for combination of independent p-values’.

If you multiply p-values together, then harmless and probable incidents rapidly appear vanishingly unlikely. Let’s say you worked in twenty hospitals, each with a harmless incident pattern: say p=0.5. If you multiply those harmless p-values, of entirely chance findings, you end up with a final p-value of 0.5 to the power of twenty, which is p < 0.000001, which is extremely, very, highly statistically significant. With this mathematical error, by his reasoning, if you change hospitals a lot, you automatically become a suspect. Have you worked in twenty hospitals? For God’s sake don’t tell the Dutch police if you have.

The figures here are ballpark, from Gerd Gigerenzer’s excellent book Reckoning with Risk.

The magician and pseudoscience debunker James Randi used to wake up every morning and write on a card in his pocket: ‘I, James Randi, will die today’, followed by the date and his signature. Just in case, he has recently explained, he really did, by some completely unpredictable accident.

Health Scares

In the previous chapter we looked at individual cases: they may have been egregious, and in some respects absurd, but the scope of the harm they can do is limited. We have already seen, with the example of Dr Spock’s advice to parents on how their babies should sleep, that when your advice is followed by a very large number of people, if you are wrong, even with the best of intentions, you can do a great deal of harm, because the effects of modest tweaks in risk are magnified by the size of the population changing its behaviour.

It’s for this reason that journalists have a special responsibility, and that’s also why we will devote the last chapter of this book to examining the processes behind two very illustrative scare stories: the MRSA swabs hoax, and MMR. But as ever, as you know, we are talking about much more than just those two stories, and there will be many distractions along the way.

The Great MRSA Hoax

There are many ways in which journalists can mislead a reader with science: they can cherry-pick the evidence, or massage the statistics; they can pit hysteria and emotion against cold, bland statements from authority figures. The MRSA stings of 2005 come as close to simply ‘making stuff up’ as anything I’ve stumbled on so far.

I first worked out what was going on when I got a phone call from a friend who works as an undercover journalist for television.‘ I just got a job as a cleaner to take some MRSA swabs for my filthy hospital superbug scandal,’ he said, ‘but they all came back negative. What am I doing wrong?’ Happy to help, I explained that MRSA doesn’t survive well on windows and doorknobs. The stories he had seen elsewhere were either rigged or rigged. Ten minutes later he rang back, triumphant: he had spoken to a health journalist from a well-known tabloid, and she had told him exactly which lab to use: ‘the lab that always gives positive results’ were the words she used, and it turned out to be Northants-based Chemsol Consulting, run by a man called Dr Christopher Malyszewicz. If you have ever seen an undercover MRSA superbug positive swab scandal, it definitely came from here. They all do.

Microbiologists at various hospitals had been baffled when their institutions fell victim to these stories. They took swabs from the same surfaces, and sent them to reputable mainstream labs, including their own: but the swabs grew nothing, contrary to Chemsol’s results. An academic paper by eminent microbiologists describing this process in relation to one hospital – UCLH – was published in a peer-reviewed academic journal, and loudly ignored by everyone in the media.

Before we go any further, we should clarify one thing, and it relates to the whole of this section on health scares: it is very reasonable to worry about health risks, and to check them out carefully. Authorities are not to be trusted, and in this specific case, plenty of hospitals aren’t as clean as we’d like them to be. Britain has more MRSA than many other countries, and this could be for any number of reasons, including infection control measures, cleanliness, prescribing patterns, or things we’ve not thought of yet (I’m talking off the top of my head here).

But we’re looking at the issue of one private laboratory, with an awful lot of business from undercover journalists doing MRSA undercover swab stories, that seems to give an awful lot of positive results.

I decided to ring Dr Chris Malyszewicz and ask if he had any ideas why this should be.

He said he didn’t know, and suggested that the hospital microbiologists might be taking swabs from the wrong places at the wrong times. They can often be incompetent, he explained. I asked why he thought the tabloids always chose his lab (producing almost twenty articles so far, including a memorable ‘mop of death’ front page in the Sunday Mirror). He had no idea. I asked why various microbiologists had said he refused to disclose his full methods, when they only wanted to replicate his techniques in their own labs in order to understand the discrepancy. He said he’d told them everything (I suspect in retrospect he was so confused that he believed this to be true). He also mispronounced the names of some very common bacteria.

It was at this point that I asked Dr Malyszewicz about his qualifications. I don’t like to critique someone’s work on the basis of who they are, but it felt like a fair question under the circumstances. On the telephone, being entirely straight, he just didn’t feel like a man with the intellectual horsepower necessary to be running a complex microbiology laboratory.

He told me he had a BSc from Leicester University. Actually it’s from Leicester Polytechnic. He told me he has