Sometimes you just want a story to be true. ‘Brisk walks are better than going to the gym,’ promised the Daily Mail. I love walking and hate gyms. I downloaded the study with glee and started to read.
The paper, based on analysis of data from several years’ of the annual English Health Survey, was heavy on statistical analysis and calculations. The section that began: ‘wit = βXi t + γ Pi t + θ Sk + λτt + i t’ did rather lose me. But after a while, I noticed something.
The researchers had looked for the impact of any type of regular physical activity, and its correlation with body mass index and waist circumference. They’d then looked at four different types of activity, and how they correlated with these measurements. Unsurprisingly, people who did more physical activity had lower BMIs and smaller waists than those who did less. Looking at the data separately, the correlation was strongest (for BMI measurements in men and women, and for waist measurements in women) with walking. The category ‘sports/exercise’, which included running, cycling, swimming, gym work-outs as well as playing sport, also showed a correlation, but it was less strong.
The researchers presented their results as showing that walking was more likely to keep weight off than sports/exercise. But I couldn’t see any indication that they had directly compared the two types of activity. I checked with colleagues, who agreed. As far as we could tell, they’d taken the difference between weight measurements of people who regularly did these activities, compared to people who did not. The difference was larger for walking, but that doesn’t automatically mean walking is better. More people walk regularly, so the correlation was likely to be stronger. Had they compared the weight measurements directly, controlling for other factors and taking into account the play of chance, the difference between walking and sport/exercise might have evaporated.
False comparisons are one of the first tricks I was warned about when I started looking at health research in the 1990s. Then, the chances were the paper had come from a pill maker, keen to show that drug X was better than drug Y, without doing a direct and fair comparison between the two. Changes from baseline, or relative differences from placebo group response, were proffered instead. Wise colleagues warned against taking these at face value, because of the difficulty in telling whether they are statistically significant.
In this case, unsurprisingly, most UK journalists took the researchers’ conclusions at face value. A combination of an impenetrable paper and an irresistible message.
Here’s my attempt to explain it for NHS Choices’ Behind the Headlines.