How do we find out what works? Not just in medicine, although that’s my usual field, but in life, for most people, most of the time.
The short, sciencey answer is to run an experiment to find out. So, for example, if we want to know whether TV makes people happy, we might do an experiment where half of a group of people watch their usual amount of TV for a month, and record each day how happy they feel on a scale of 1 to 10. The other half do the same, but have their TVs taken away from them at the start of the experiment.
What could stop this from working? Well, clearly some people like watching TV much more than others. And some people never watch TV anyway. Some people feel miserable if they miss Eastenders, while others only turn it on during Wimbledon fortnight. So a lot depends on who is picked for each group.
If we pick all the people who love watching TV, and put them in one group, we’ll get a completely skewed result. If they all go in the group that has their TV taken away, they may be completely miserable, making it look as if watching TV makes you happy. On the other hand, if we put all the people who never watch TV in that group, they’ll not notice any difference. You’ll probably get a result where both groups seem equally happy.
The key is to get a balance of people in both groups. You could do this by asking everyone how much TV they watch, then carefully balancing the two groups. But that’s laborious, and still might not reflect how much pleasure people get from their TV-watching. Far better is to randomly assign people to one group or the other, for example by getting them to draw numbers from a bag. Everyone with an even number has their TV, everyone with an odd number loses it for a month.
If you have a big enough group to start with, you should have roughly equal numbers of TV addicts and non-watchers in each group. That way, you’ll find out what TV deprivation or TV watching does to average happiness levels.
It’s called randomisation, and it’s one of the most important parts of running a decent scientific study. Yet it’s often overlooked, with the attention being grabbed by the size of the study, or whether it included a placebo arm, or whether the participants were blinded to the outcome. All these things are important, but randomisation can make or break a study.
A good example of why it’s important happened about a decade ago, when the first results came in from randomised trials of hormone replacement therapy (HRT). Previously, it looked like women taking HRT lived longer and were less likely to have heart attacks or strokes. The trouble was, most of the women who’d chosen HRT for themselves i the early years had things in common. They were more likely to go to the doctor with menopausal symptoms. They were better educated – they’d heard about HRT – and more health-conscious. They were likely to be better-off, too, without concerns about health insurance.
The non-randomly selected studies compared this well-educated, health-conscious, richer group of women, with women who didn’t choose HRT. It’s not surprising that they were overall healthier women, and so the results looked good for HRT. As soon as trials started with randomisation – which meant a random distribution of all these characteristics between those who took HRT and those who didn’t take it – the advantage melted away. Not only that, but an increase in risk of heart attack and stroke, as well as of breast cancer, became clear. Lack of randomisation had given the exact opposite result to the truth.
Randomisation is hard to do and is rarely attempted outside medicine. Which makes it very hard to know, for example, how to improve children’s education. Do grammar schools get good results because their teaching methods are better, or because they are chosen by well-off families who believe in education? A randomised trial might show us, but it’s hard to imagine parents agreeing to put their children into trials. The trouble is, without randomisation, all pupils are in a big, uncontrolled trial, which is unlikely to tell us anything useful.
The same is true of transport, housing, practically any area where we don’t really know which methods produce the best results. Governments wishing to test new policies tend to run ‘pilot’ schemes, which are designed to show that something they’ve decided they want to do works. It would be a brave politician who agreed to a properly-run scientific trial of his or her favourite policy, to see how the results compared to a rival policy.
So I’m looking forward to the Evidence-based Policy discussion at the Wellcome Collection next week. Only one caveat though – three good scientists, but no politicians on the panel.
Image: From Jeremy Brooks’ photostream on Flickr, with CCL.