There's no gain to anyone in publishing flawed work in serious journals, argues Peter Collins
Scientists with exciting findings sometimes go to the press before checking their research with fellow scientists. A vivid current example is Arpad Pusztai. In an experiment that has now become infamous, Dr Pusztai fed rats potatoes that had been genetically modified to contain a lectin gene from snowdrops. This seemed to affect the rats' weight and their immune systems. Pusztai told a World in Action team this was because of the genetic modification rather than the lectin or another factor.
The result was huge controversy and alarm about the risks to people of genetically engineered food. The Royal Society, disturbed that a one-sided debate was raging in the media on the back of unvalidated data, set itself the task of reviewing as much of Pusztai's data as it could obtain, which it sent to six anonymous referees, whose responses it then endorsed.
The society concluded Pusztai's work was "flawed in many aspects of design, execution and analysis and no conclusions should be drawn from it". Moreover, even if the experiments had been properly devised and carried out "it would be unjustifiable to draw from them general conclusions about whether genetically modified foods are harmful to human beings".
At this stage, the potato work had still not completed the process of conventional peer review and publication. A paper had, in fact, been submitted to the medical journal The Lancet towards the end of 1998, but it was not very good. Indeed, it had to be revised three times before The Lancet would accept it. So publication was not going to be straightforward.
The Lancet's editor had a problem. He had gone out of his way to support anti-GM sentiment and to criticise those who were critical of Pusztai. He had slated the Royal Society for its "breathtaking impertinence" in daring to review Pusztai's unpublished work (The Lancet, May 29, 1999). He had given space to Pusztai to defend himself (The Lancet, August 21, 1999) and so he was more or less committed to publishing the paper by Stanley Ewen and Pusztai when it was finished (The Lancet, October 16, 1999). He resolved the dilemma by accompanying the paper with an editorial defending his decision to publish, and renewing his attack on Pusztai's critics.
Fellows of the Royal Society read last week's Lancet paper very closely. The society's original view stands: the newly published experiments, based on the work reviewed in May, cannot support the interpretation being put on them. The claimed effects were slight, the sample size was small and the many dietary variables were so poorly controlled that to attribute the effects seen to any one of them is untenable.
So what was achieved by publishing this paper? We have been calling for the data to be made public ever since the story first broke. So it is useful that some data, at least, is now generally available.
But it is not just data that has been published: the Lancet paper also includes a couple of speculative assertions - "the stimulatory effect ... was mainly due to the expression of the GNA transgene in the potato", and "the possibility that a plant vector ... can affect the mucosa of the gastrointestinal tract ... may also apply to GM plants containing similar constructs".
The Lancet's editor stressed that Ewen and Pusztai's data was not generalisable and that publication did not vindicate Pusztai's earlier claims, but he still allowed the speculations to be presented as serious conclusions in a scientific paper.
The trouble is that The Lancet has a wide readership and many readers without specialist knowledge may be tempted to take the conclusions on trust. For such readers, the fact of publication conveys a degree of credibility to the paper as a whole. That is one of the consequences of peer review, which is why Aaron Klug, president of the Royal Society, like some of The Lancet's peer reviewers, argued strongly that publication was not a good idea.
The Lancet editor lectures us on the "new, and apparently unwelcome, dialogue of accountability that needs to be forged between scientists and the public". The dialogue is neither new nor unwelcome. But it requires exercise of critical judgement, not suspension of it. Neither science nor society are served by publishing flawed work in serious journals.
Peter Collins is director of science policy, The Royal Society.