Peer review system 'leads to good research going unpublished'

The first experiment to fully model the dynamics of academia’s fiercely competitive peer review system has concluded that it increases creative diversity and innovation, but incentivises researchers to mark their rivals down, meaning that high-quality research goes unpublished.

It adds to doubts over the current system, with one of the authors of the new paper urging journals to experiment further with post-publication peer review, where papers are assessed openly after being made public.

To test what happens when creators assess the work of their rivals – in an attempt to model the peer review system – researchers asked 144 participants in Zurich to come up with works of art on a computer, which were then judged by their fellow participants. This was repeated 30 times, and depending on how well they were rated, participants won up to SFr80 (£61).

Using sliders, participants were able to modify a picture of a face, or transform it beyond recognition into an abstract image. 

Stefano Balietti, a postdoctoral student at the Network Science Institute at Northeastern University in Boston and one of the authors of the paper, said that they “gradually moved towards more abstract art, a bit like what happened in the history of art”.

The researchers then sourced tens of thousands of online reviews of the artworks to come up with as objective a measure of their quality as possible.

They found that when participants were in competition with each other, and had to get high scores from their competitors to win money, they produced more diverse, innovative and abstract work in an attempt to mark themselves out from competitors.

“But more diverse does not mean better,” cautioned Dr Balietti. The experiment also discovered that competition between the participants made them likely to behave “strategically”, marking down rivals to boost their own chances – a common fear about the peer review process.

Competition led to higher rejection rates but did not mean that higher quality art was published, the study found. Dr Balietti said this was most likely because game-playing among participants meant that otherwise good artwork was penalised.

“Our results could explain why many ground-breaking studies in science end up in lower-tier journals,” concludes “Peer review and competition in the Art Exhibition Game”, published in the journal PNAS.

The results feed in to the debate about rejection rates, which at some high-profile journals can reach more than 90 per cent, making competition for inclusion intense.

Dr Balietti said that the results of his experiment indicated that academia faces a trade-off: competition leads to a greater diversity of papers but also means that otherwise good research may not be published.  

When tackling a problem such as climate change, he said, “more competition might not be the best solution. We might by mistake reject the one good solution.”

The paper argues for “career schemes that tolerate early failure and focus on long-term success” as a way to encourage innovation in research. Dr Balietti also argued that more journals should “experiment” with post-publication peer review.

david.matthews@tesglobal.com