Bad Science

experiences, for the reasons described above: they might be mistaking the placebo effect for a real effect, or mistaking a chance finding for a real one. Even if we had one genuine, unambiguous and astonishing case of a person getting better from terminal cancer, we’d still be careful about using that one person’s experience, because sometimes, entirely by chance, miracles really do happen. Sometimes, but not very often.

Over the course of many years, a team of Australian oncologists followed 2,337 terminal cancer patients in palliative care. They died, on average, after five months. But around 1 per cent of them were still alive after five years. In January 2006 this study was reported in the Independent, bafflingly, as:

‘Miracle’ Cures Shown to Work

Doctors have found statistical evidence that alternative treatments such as special diets, herbal potions and faith healing can cure apparently terminal illness, but they remain unsure about the reasons.

But the point of the study was specifically not that there are miracle cures (it didn’t look at any such treatments, that was an invention by the newspaper). Instead, it showed something much more interesting: that amazing things simply happen sometimes: people can survive, despite all the odds, for no apparent reason. As the researchers made clear in their own description, claims for miracle cures should be treated with caution, because ‘miracles’ occur routinely, in 1 per cent of cases by their definition, and without any specific intervention. The lesson of this paper is that we cannot reason from one individual’s experience, or even that of a handful, selected out to make a point.

So how do we move on? The answer is that we take lots of individuals, a sample of patients who represent the people we hope to treat, with all of their individual experiences, and count them all up. This is clinical academic medical research, in a nutshell, and there’s really nothing more to it than that: no mystery, no ‘different paradigm’, no smoke and mirrors. It’s an entirely transparent process, and this one idea has probably saved more lives, on a more spectacular scale, than any other idea you will come across this year.

It is also not a new idea. The first trial appears in the Old Testament, and interestingly, although nutritionism has only recently become what we might call the ‘bollocks du jour’, it was about food. Daniel was arguing with King Nebuchadnezzar’s chief eunuch over the Judaean captives’ rations. Their diet was rich food and wine, but Daniel wanted his own soldiers to be given only vegetables. The eunuch was worried that they would become worse soldiers if they didn’t eat their rich meals, and that whatever could be done to a eunuch to make his life worse might be done to him. Daniel, on the other hand, was willing to compromise, so he suggested the first ever clinical trial:

And Daniel said unto the guard … ‘Submit us to this test for ten days. Give us only vegetables to eat and water to drink; then compare our looks with those of the young men who have lived on the food assigned by the King and be guided in your treatment of us by what you see.’

The guard listened to what they said and tested them for ten days. At the end of ten days they looked healthier and were better nourished than all the young men who had lived on the food assigned them by the King. So the guard took away the assignment of food and the wine they were to drink and gave them only the vegetables.

Daniel 1: 1–16.

To an extent, that’s all there is to it: there’s nothing particularly mysterious about a trial, and if we wanted to see whether homeopathy pills work, we could do a very similar trial. Let’s flesh it out. We would take, say, two hundred people going to a homeopathy clinic, divide them randomly into two groups, and let them go through the whole process of seeing the homeopath, being diagnosed, and getting their prescription for whatever the homeopath wants to give them. But at the last minute, without their knowledge, we would switch half of the patients’ homeopathic sugar pills, giving them dud sugar pills, that have not been magically potentised by homeopathy. Then, at an appropriate time later, we could measure how many in each group got better.

Speaking with homeopaths, I have encountered a great deal of angst about the idea of measuring, as if this was somehow not a transparent process, as if it forces a square peg into a round hole, because ‘measuring’ sounds scientific and mathematical. We should pause for just a moment and think about this clearly. Measuring involves no mystery, and no special devices. We ask people if they feel better, and count up the answers.

In a trial – or sometimes routinely in outpatients’ clinic – we might ask people to measure their knee pain on a scale of one to ten every day, in a diary. Or to count up the number of pain-free days in a week. Or to measure the effect their fatigue has had on their life that week: how many days they’ve been able to get out of the house, how far they’ve been able to walk, how much housework they’ve been able to do. You can ask about any number of very simple, transparent, and often quite subjective things, because the business of medicine is improving lives, and ameliorating distress.

We might dress the process up a bit, to standardise it, and allow our results to be compared more easily with other research (which is a good thing, as it helps us to get a broader understanding of a condition and its treatment). We might use the ‘General Health Questionnaire’, for example, because it’s a standardised ‘tool’; but for all the bluster, the ‘GHQ-12’, as it is known, is just a simple list of questions about your life and your symptoms.

If anti-authoritarian rhetoric is your thing, then bear this in mind: perpetrating a placebo-controlled trial of an accepted treatment – whether it’s an alternative therapy or any form of medicine – is an inherently subversive act. You undermine false certainty, and you deprive doctors, patients and therapists of treatments which previously pleased them.

There is a long history of upset being caused by trials, in medicine as much as anywhere, and all kinds of people will mount all kinds of defences against them. Archie Cochrane, one of the grandfathers of evidence-based medicine, once amusingly described how different groups of surgeons were each earnestly contending that their treatment for cancer was the most effective: it was transparently obvious to them all that their own treatment was the best. Cochrane went so far as to bring a collection of them together in a room, so that they could witness each other’s dogged but conflicting certainty, in his efforts to persuade them of the need for trials. Judges, similarly, can be highly resistant to the notion of trialling different forms of sentence for heroin users, believing that they know best in each individual case. These are recent battles, and they are in no sense unique to the world of homeopathy.

So, we take our group of people coming out of a homeopathy clinic, we switch half their pills for placebo pills, and we measure who gets better. That’s a placebo-controlled trial of homeopathy pills, and this is not a hypothetical discussion: these trials have been done on homeopathy, and it seems that overall, homeopathy does no better than placebo.

And yet you will have heard homeopaths say that there are positive trials in homeopathy; you may even have seen specific ones quoted. What’s going on here? The answer is fascinating, and takes us right to the heart of evidence-based medicine. There are some trials which find homeopathy to perform better than placebo, but only some, and they are, in general, trials with ‘methodological flaws’. This sounds technical, but all it means is that there are problems in the way the trials were performed, and those problems are so great that they mean the trials are less ‘fair tests’ of a treatment.

The alternative therapy literature is certainly riddled with incompetence, but flaws in trials are actually very common throughout medicine. In fact, it would be fair to say that all research has some ‘flaws’, simply because every trial will involve a compromise between what would be ideal, and what is practical or cheap. (What sets the CAM literature apart is, in some respects, the interpretation: medics sometimes know if they’re quoting duff papers, and describe the flaws, whereas homeopaths tend to be uncritical of anything positive.)

That is why it’s important that research is always published, in full, with its methods and results available for scrutiny. This is a recurring theme in this book, and it’s important, because when people make claims based upon their research, we need to be able to decide for ourselves how big the ‘methodological flaws’ were, and come to our own judgement about whether the results are reliable, whether theirs was a ‘fair test’. The things that stop a trial from being fair are, once you know about them, blindingly obvious.

Blinding

One important feature of a good trial is that neither the experimenters nor the patients know if they got the homeopathy sugar pill or the simple placebo sugar pill, because we want to be sure that any difference we measure is the result of the difference between the pills, and not of people’s expectations or biases. If the researchers knew which of their beloved patients were having the real and which the placebo pills, they might give the game away –