Bad Science

cut-off between normal and abnormal values?

All I am doing, I should be clear, is taking the feted media nutritionists at their own word: they present themselves as men and women of science, fill their columns, TV shows and books with references to scientific research. I am subjecting their claims to the exact same level of very basic, uncomplicated rigour that I would deploy for any new theoretical work, any drug company claim and pill marketing rhetoric, and so on.

It’s not unreasonable to use surrogate outcome data, as they do, but those who are in the know are always circumspect. We’re interested in early theoretical work, but often the message is: ‘It might be a bit more complicated than that …’. You’d only want to accord a surrogate outcome any significance if you’d read everything on it yourself, or if you could be absolutely certain that the person assuring you of its validity was extremely capable, and was giving a sound appraisal of all the research in a given field, and so on.

Similar problems arise with animal data. Nobody could deny that this kind of data is valuable in the theoretical domain, for developing hypotheses, or suggesting safety risks, when cautiously appraised. But media nutritionists, in their eagerness to make lifestyle claims, are all too often blind to the problems of applying these isolated theoretical nuggets to humans, and anyone would think they were just trawling the internet looking for random bits of science to sell their pills and expertise (imagine that). Both the tissue and the disease in an animal model, after all, may be very different to those in a living human system, and these problems are even greater with a lab-dish model. Giving unusually high doses of chemicals to animals can distort the usual metabolic pathways, and give misleading results – and so on. Just because something can upregulate or downregulate something in a model doesn’t mean it will have the effect you expect in a person – as we will see with the stunning truth about antioxidants.

And what about turmeric, which we were talking about before I tried to show you the entire world of applying theoretical research in this tiny grain of spice? Well, yes, there is some evidence that curcumin, a chemical in turmeric, is highly biologically active, in all kinds of different ways, on all kinds of different systems (there are also theoretical grounds for believing that it may be carcinogenic, mind you). It’s certainly a valid target for research.

But for the claim that we should eat more curry in order to get more of it, that ‘recent research’ has shown it is ‘highly protective against many forms of cancer, especially of the prostate’, you might want to step back and put the theoretical claims in the context of your body. Very little of the curcumin you eat is absorbed. You have to eat a few grams of it to reach significant detectable serum levels, but to get a few grams of curcumin, you’d have to eat 100g of turmeric: and good luck with that. Between research and recipe, there’s a lot more to think about than the nutritionists might tell you.

Cherry-picking

The idea is to try and give all the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.

Richard P. Feynman

There have been an estimated fifteen million medical academic articles published so far, and 5,000 journals are published every month. Many of these articles will contain contradictory claims: picking out what’s relevant – and what’s not – is a gargantuan task. Inevitably people will take shortcuts. We rely on review articles, or on meta-analyses, or textbooks, or hearsay, or chatty journalistic reviews of a subject.

That’s if your interest is in getting to the truth of the matter. What if you’ve just got a point to prove? There are few opinions so absurd that you couldn’t find at least one person with a PhD somewhere in the world to endorse them for you; and similarly, there are few propositions in medicine so ridiculous that you couldn’t conjure up some kind of published experimental evidence somewhere to support them, if you didn’t mind it being a tenuous relationship, and cherry-picked the literature, quoting only the studies that were in your favour.

One of the great studies of cherry-picking in the academic literature comes from an article about Linus Pauling, the great-grandfather of modern nutritionism, and his seminal work on vitamin C and the common cold. In 1993 Paul Knipschild, Professor of Epidemiology at the University of Maastricht, published a chapter in the mighty textbook Systematic Reviews: he had gone to the extraordinary trouble of approaching the literature as it stood when Pauling was working, and subjecting it to the same rigorous systematic review that you would find in a modern paper.

He found that while some trials did suggest that vitamin C had some benefits, Pauling had selectively quoted from the literature to prove his point. Where Pauling had referred to some trials which seriously challenged his theory, it was to dismiss them as methodologically flawed: but as a cold examination showed, so too were papers he quoted favourably in support of his own case.

In Pauling’s defence, his was an era when people knew no better, and he was probably quite unaware of what he was doing: but today cherry-picking is one of the most common dubious practices in alternative therapies, and particularly in nutritionism, where it seems to be accepted essentially as normal practice (it is this cherry- picking, in reality, which helps to characterise what alternative therapists conceive of rather grandly as their ‘alternative paradigm’). It happens in mainstream medicine also, but with one crucial difference: there it is recognised as a major problem, and hard work has been done to derive a solution.

That solution is a process called ‘systematic review’. Instead of just mooching around online and picking out your favourite papers to back up your prejudices and help you sell a product, in a systematic review you have an explicit search strategy for seeking out data (openly described in your paper, even including the search terms you used on databases of research papers), you tabulate the characteristics of each study you find, you measure – ideally blind to the results – the methodological quality of each one (to see how much of a ‘fair test’ it is), you compare alternatives, and then finally you give a critical, weighted summary.

This is what the Cochrane Collaboration does on all the healthcare topics that it can find. It even invites people to submit new clinical questions that need an answer. This careful sifting of information has revealed huge gaps in knowledge, it has revealed that ‘best practices’ were sometimes murderously flawed, and simply by sifting methodically through preexisting data, it has saved more lives than you could possibly imagine. In the nineteenth century, as the public-health doctor Muir Gray has said, we made great advances through the provision of clean, clear water; in the twenty-first century we will make the same advances through clean, clear information. Systematic reviews are one of the great ideas of modern thought. They should be celebrated.

Problematising antioxidants

We have seen the kinds of errors made by those in the nutritionism movement as they strive to justify their more obscure and technical claims. What’s more fun is to take our new understanding and apply it to one of the key claims of the nutritionism movement, and indeed to a fairly widespread belief in general: the claim that you should eat more antioxidants.

As you now know, there are lots of ways of deciding whether the totality of research evidence for a given claim stacks up, and it’s rare that one single piece of information clinches it. In the case of a claim about food, for example, there are all kinds of different things we might look for: whether it is theoretically plausible, whether it is backed up by what we know from observing diets and health, whether it is supported by ‘intervention trials’ where we give one group one diet and another group a different one, and whether those trials measured real-world outcomes, like ‘death’, or a surrogate outcome, like a blood test, which is only hypothetically related to a disease.

My aim here is by no means to suggest that antioxidants are entirely irrelevant to health. If I had a T-shirt slogan for this whole book it would be: ‘I think you’ll find it’s a bit more complicated than that’. I intend, as they say, to ‘problematise’ the prevailing nutritionist view on antioxidants, which currently lags only about twenty years behind the research evidence.

From an entirely theoretical perspective, the idea that antioxidants are beneficial for health is an attractive one. When I was a medical student – not so long ago – the most popular biochemistry textbook was called Stryer. This enormous book is filled with complex interlocking flow charts of how chemicals – which is what you are made of – move through the body. It shows how different enzymes break down food into its constituent molecular elements, how these are absorbed, how they are reassembled into new larger molecules that your body needs to build muscles, retina, nerves, bone, hair, membrane, mucus, and everything else that you’re made of; how the various forms of fats are broken down, and reassembled into new forms of fat; or how different forms of molecule – sugar, fat, even alcohol – are broken down gradually, step by step, to release energy, and how that energy is