Bad Science

into the pre-existing ‘miracle-cure-hidden-scare’ template.

I would go further, and argue that science itself works very badly as a news story: it is by its very nature a subject for the ‘features’ section, because it does not generally move ahead by sudden, epoch-making breakthroughs. It moves ahead by gradually emergent themes and theories, supported by a raft of evidence from a number of different disciplines on a number of different explanatory levels. Yet the media remain obsessed with ‘new breakthroughs’.

It’s quite understandable that newspapers should feel it’s their job to write about new stuff. But if an experimental result is newsworthy, it can often be for the same reasons that mean it is probably wrong: it must be new, and unexpected, it must change what we previously thought; which is to say, it must be a single, lone piece of information which contradicts a large amount of pre-existing experimental evidence.

There has been a lot of excellent work done, much of it by a Greek academic called John Ioannidis, demonstrating how and why a large amount of brand-new research with unexpected results will subsequently turn out to be false. This is clearly important in the application of scientific research to everyday work, for example in medicine, and I suspect most people intuitively understand that: you would be unwise to risk your life on a single piece of unexpected data that went against the grain.

In the aggregate, these ‘breakthrough’ stories sell the idea that science – and indeed the whole empirical world view – is only about tenuous, new, hotly contested data and spectacular breakthroughs. This reinforces one of the key humanities graduates’ parodies of science: as well as being irrelevant boffinry, science is temporary, changeable, constantly revising itself, like a transient fad. Scientific findings, the argument goes, are therefore dismissible.

While this is true at the bleeding edges of various research fields, it’s worth bearing in mind that Archimedes has been right about why things float for a couple of millennia. He also understood why levers work, and Newtonian physics will probably be right about the behaviour of snooker balls forever. But somehow this impression about the changeability of science has bled through to the core claims. Anything can be rubbished.

But that is all close to hand-waving. We should now look at how the media cover science, unpick the real meanings behind the phrase ‘research has shown’, and, most importantly of all, examine the ways in which the media repeatedly and routinely misrepresent and misunderstand statistics.

‘Research has shown …’

The biggest problem with science stories is that they routinely contain no scientific evidence at all. Why? Because papers think you won’t understand the ‘science bit’, so all stories involving science must be dumbed down, in a desperate bid to seduce and engage the ignorant, who are not interested in science anyway (perhaps because journalists think it is good for you, and so should be democratised).

In some respects these are admirable impulses, but there are certain inconsistencies I can’t help noticing. Nobody dumbs down the finance pages. I can barely understand most of the sports section. In the literature pull- out, there are five-page-long essays which I find completely impenetrable, where the more Russian novelists you can rope in the cleverer everybody thinks you are. I do not complain about this: I envy it.

If you are simply presented with the conclusions of a piece of research, without being told what was measured, how, and what was found – the evidence – then you are simply taking the researchers’ conclusions at face value, and being given no insight into the process. The problems with this are best explained by a simple example.

Compare the two sentences ‘Research has shown that black children in America tend to perform less well in IQ tests than white children’ and ‘Research has shown that black people are less intelligent than white people.’ The first tells you about what the research found: it is the evidence. The second tells you the hypothesis, somebody’s interpretation of the evidence: somebody who, you will agree, doesn’t know much about the relationship between IQ tests and intelligence.

With science, as we have seen repeatedly, the devil is in the detail, and in a research paper there is a very clear format: you have the methods and results section, the meat, where you describe what was done, and what was measured; and then you have the conclusions section, quite separately, where you give your impressions, and mesh your findings with those of others to decide whether they are compatible with each other, and with a given theory. Often you cannot trust researchers to come up with a satisfactory conclusion on their results – they might be really excited about one theory – and you need to check their actual experiments to form your own view. This requires that news reports are about published research which can, at least, be read somewhere. It is also the reason why publication in full – and review by anyone in the world who wants to read your paper – is more important than ‘peer review’, the process whereby academic journal articles are given the once-over by a few academics working in the field, checking for gross errors and the like.

In the realm of their favourite scares, there is a conspicuous over-reliance by newspapers on scientific research that has not been published at all. This is true of almost all of the more recent headline stories on new MMR research, for example. One regularly quoted source, Dr Arthur Krigsman, has been making widely reported claims for new scientific evidence on MMR since 2002, and has yet to publish his work in an academic journal to this day, six years later. Similarly, the unpublished ‘GM potato’ claims of Dr Arpad Pusztai that genetically modified potatoes caused cancer in rats resulted in ‘Frankenstein food’ headlines for a whole year before the research was finally published, and could be read and meaningfully assessed. Contrary to the media speculation, his work did not support the hypothesis that GM is injurious to health (this doesn’t mean it’s necessarily a good thing – as we will see later).

Once you become aware of the difference between the evidence and the hypothesis, you start to notice how very rarely you get to find out what any research has really shown when journalists say ‘research has shown’.

Sometimes it’s clear that the journalists themselves simply don’t understand the unsubtle difference between the evidence and the hypothesis. The Times, for example, covered an experiment which showed that having younger siblings was associated with a lower incidence of multiple sclerosis. MS is caused by the immune system turning on the body. ‘This is more likely to happen if a child at a key stage of development is not exposed to infections from younger siblings, says the study.’ That’s what The Times said.

But it’s wrong. That’s the ‘hygiene hypothesis’, that’s the theory, the framework into which the evidence might fit, but it’s not what the study showed: the study just found that having younger siblings seemed to be somewhat protective against MS. It didn’t say what the mechanism was, it couldn’t say why there was a relationship, such as whether it happened through greater exposure to infections. It was just an observation. The Times confused the evidence with hypothesis, and I am very glad to have got that little gripe out of my system.

How do the media work around their inability to deliver scientific evidence? Often they use authority figures, the very antithesis of what science is about, as if they were priests, or politicians, or parent figures. ‘Scientists today said … Scientists revealed … Scientists warned’. If they want balance, you’ll get two scientists disagreeing, although with no explanation of why (an approach which can be seen at its most dangerous in the myth that scientists were ‘divided’ over the safety of MMR). One scientist will ‘reveal’ something, and then another will ‘challenge’ it. A bit like Jedi knights.

There is a danger with authority-figure coverage, in the absence of real evidence, because it leaves the field wide open for questionable authority figures to waltz in. Gillian McKeith, Andrew Wakefield and the rest can all get a whole lot further in an environment where their authority is taken as read, because their reasoning and evidence are rarely publicly examined.

Worse than that, where there is controversy about what the evidence shows, it reduces the discussion to a slanging match, because a claim such as ‘MMR causes autism’ (or not), is only critiqued in terms of the character of the person who is making the statement, rather than the evidence they are able to present. There is no need for this, as we shall see, because people are not stupid, and the evidence is often pretty easy to understand.

It also reinforces the humanities graduate journalists’ parody of science, for which we now have all the ingredients: science is about groundless, changeable, didactic truth statements from arbitrary unelected authority figures. When they start to write about serious issues like MMR, you can see that this is what people in the media really think science is about. The next stop on our journey is inevitably going to be statistics, because this is one area that causes unique problems for the media. But first, we need to go on a brief diversion.