fudging

But how can you tell if the research literature on a given subject has been rigged? It’s a tricky problem, because you’re chasing evidence for the existence of trials you cannot see. One option is to use mathematical tools, and something called a funnel plot, one of the cleverest ideas of the last century. It’s so clever that you might need to concentrate for the next bit.

Let’s imagine that there are 30 trials on a given drug. Some are big, and more accurate. Some are small and less accurate, with more random noise. You’d expect that the big, accurate trials should all cluster together around the true finding, all giving similar results for the efficacy of a drug. Meanwhile the smaller, rubbish trials - because they are less accurate measures of the drugs efficacy - will be scattered about randomly, some showing the treatment to be better than the good big trials indicate, some showing that it is worse.

You could then plot all your trials on a graph, one dot for each trial. On the x-axis, left to right, is “how good the drug was shown to be by this trial” and on the y-axis, “how methodologically sound and large the trial was”. If there is no publication bias, you should get a triangle shape: at the top of your graph, you will see all your good-quality, accurate trials, clustered together around the true answer. At the bottom of the graph, you will see a broad smear of results, the poor quality trials showing random variation.

But if there is publication bias, you will see a distorted triangle: the small, poor-quality trials at the bottom will be smeared over to the right, because small trials with unwelcome results are much more likely to be overlooked, and dumped in desk drawers, than huge multicentre collaborative studies involving dozens of academics and tens of thousands of participants, which are almost definitely going to get published. If you get a distorted triangle, you know there are some interesting negative trials missing.

Ben Goldacre in the Guardian's Bad Science column, the rest here.