Bad Science

Telegraph Saturday magazine. Andy is, she tells us, ‘a handsome, glossy-haired hero to families of autistic children’. How are the family? ‘A likeable, lively family, the kind you would be happy to have as friends, pitted against mysterious forces who have planted bugging devices and have stolen patients’ records in “apparently inexplicable” burglaries.’ She fantasises – and I absolutely promise you I’m not making this up – about a Hollywood depiction of Wakefield’s heroic struggle, with Russell Crowe playing the lead ‘opposite Julia Roberts as a feisty single mother fighting for justice for her child’.

The evidence on MMR

So what is the evidence on the safety of MMR?

There are a number of ways to approach the evidence on the safety of a given intervention, depending on how much attention you have to give. The simplest approach is to pick an arbitrary authority figure: a doctor, perhaps, although this seems not to be appealing (in surveys people say they trust doctors the most, and journalists the least: this shows the flaw in that kind of survey).

You could take another, larger authority at face value, if there is one that suits you. The Institute of Medicine, the Royal Colleges, the NHS, and more, all came out in support of MMR, but this was apparently not sufficient to convince. You could offer information: an NHS website at mmrthefacts.nhs.uk started with the phrase ‘MMR is safe’ (literally), and allowed the reader to drill down to the detail of individual studies. But that too did little to stem the tide. Once a scare is running, perhaps every refutation can seem like an admission of guilt, drawing attention to the scare.

The Cochrane Collaboration is as blemishless as they come, and it has done a systematic review of the literature on MMR, concluding that there was no evidence that it is unsafe (although it didn’t appear until 2005). This reviewed the data the media had systematically ignored: what was in it?

If we are to maintain the moral high ground, there are a few things we need to understand about evidence. Firstly, there is no single golden study which proves that MMR is safe (although the evidence to say it is dangerous was exceptionally poor). There is, for example, no randomised controlled trial. We are presented instead with a huge morass of data, from a number of different studies, all of which are flawed in their own idiosyncratic ways for reasons of cost, competence and so on. A common problem with applying old data to new questions is that these papers and datasets might have a lot of useful information, which was collected very competently to answer the questions which the researchers were interested in at the time, but which isn’t perfect for your needs. It’s just, perhaps, pretty good.

Smeeth et al., for example, did something called a ‘casecontrol’ study, using the GP Research Database. This is a common type of study, where you take a bunch of people with the condition you’re looking at (‘autism’), and a bunch of people without it, then look to see if there is any difference in how much each group was exposed to the thing you think might be causing the condition (‘MMR’).

If you care who paid for the study – and I hope you’ve become a bit more sophisticated than that by now – it was funded by the Medical Research Council. They found around 1,300 people with autism, and then got some ‘controls’: random people who did not have autism, but with the same age, sex, and general practice. Then they looked to see if vaccination was any more common in the people with autism, or the controls, and found no difference between the two groups. The same researchers also did a systematic review of similar studies in the United States and Scandinavia, and again, pooling the data, found no link between MMR and autism.

There is a practical problem with this kind of research, of course, which I would hope you might spot: most people do get the MMR vaccine, so the individuals you’re measuring who didn’t get the vaccine might be unusual in other ways – perhaps their parents have refused the vaccine for ideological or cultural reasons, or the child has a pre-existing physical health problem – and those factors might themselves be related to autism. There’s little you can do in terms of study design about this potential ‘confounding variable’, because as we said, you’re not likely to do a randomised controlled trial in which you randomly don’t give children vaccines: you just throw the result into the pot with the rest of the information, in order to reach your verdict. As it happens, Smeeth et al. went to great lengths to make sure their controls were representative. If you like, you can read the paper and decide if you agree.

So ‘Smeeth’ was a ‘casecontrol study’, where you compare groups which had the outcome or not, and look at how common the exposure was in each group. In Denmark, Madsen et al. did the opposite kind of study, called a ‘cohort study’: you compare groups which had the exposure or not, in order to see whether there is any variation in the outcome. In this specific case, then, you take two groups of people, who either had MMR or didn’t, and then check later to see if the rate of autism is any different between the two groups.

This study was big – very big – and included all the children born in Denmark between January 1991 and December 1998. In Denmark there is a system of unique personal identification numbers, linked to vaccination registers and information about the diagnosis of autism, which made it possible to chase up almost all the children in the study. This was a pretty impressive achievement, since there were 440,655 children who were vaccinated, and 96,648 who were unvaccinated. No difference was found between vaccinated and unvaccinated children, in the rates of autism or autistic spectrum disorders, and no association between development of autism and age at vaccination.

Anti-MMR campaigners have responded to this work by saying that only a small number of children are harmed by the vaccine, which seems to be inconsistent with their claims that MMR is responsible for a massive upswing in diagnoses of autism. In any case, if a vaccine caused an adverse reaction in a very small number of people, that would be no surprise – it would be no different from any other medical intervention (or, arguably, any human activity), and there would be, surely, no story.

As with all studies, there are problems with this huge study. The follow-up of diagnostic records ends one year (31 December 1999) after the last day of admission to the cohort: so, because autism comes on after the age of one year, the children born later in the cohort would be unlikely to have shown up with autism by the end of the follow-up period. But this is flagged up in the study, and you can decide whether you think it undermines its overall findings. I don’t think it’s much of a problem. That’s my verdict, and I think you might agree that it’s not a particularly foolish one. It did run from January 1991, after all.

This is the kind of evidence you will find in the Cochrane review, which found, very simply, that ‘existing evidence on the safety and effectiveness of MMR vaccine supports current policies of mass immunisation aimed at global measles eradication in order to reduce morbidity and mortality associated with mumps and rubella’.

It also contained multiple criticisms of the evidence it reviewed, which, bizarrely, has been seized upon by various commentators to claim that there was some kind of stitch-up. The review was heading towards a conclusion that MMR was risky, they say, if you read the content, but then, out of nowhere, it produced a reassuring conclusion, doubtless because of hidden political pressure.

The Daily Mail’s Melanie Phillips, a leading light of the antivaccination movement, was outraged by what she thought she had found: ‘It said that no fewer than nine of the most celebrated studies that have been used against [Andrew Wakefield] were unreliable in the way they were constructed.’ Of course it did. I’m amazed it wasn’t more. Cochrane reviews are intended to criticise papers.

Scientific ‘evidence’ in the media

But the newspapers in 2002 had more than just worried parents. There was a smattering of science to keep things going: you will remember computer-generated imagery of viruses and gut walls, perhaps, and stories about laboratory findings. Why have I not mentioned those?

For one thing, these important scientific findings were being reported in newspapers and magazines, and at meetings, in fact anywhere except proper academic journals where they could be read and carefully appraised. In May, for example, Wakefield ‘exclusively revealed’ that ‘more than 95 per cent of those who had the virus in their gut had MMR as their only documented exposure to measles’. He doesn’t appear to have revealed this in a peer- reviewed academic journal, but in a weekend colour supplement.

Other people started popping up all over the place, claiming to have made some great finding, but never publishing their research in proper, peer-reviewed academic journals. A pharmacist in Sunderland called Dr Paul Shattock was reported on the Today programme, and in several national newspapers, to have identified a distinct subgroup of children with autism resulting from MMR. Dr Shattock is very active on anti- immunisation websites, but he still doesn’t seem to have got round to publishing this important work years later, even though the Medical Research Council suggested in 2002 that he should ‘publish his research and come forward