Time and experience are no friends of peer review

A study billed as the most extensive conducted of peer reviewers has exposed an unexpected flaw in the process.

While it might seem safe to assume that experience brings benefits, the research by academics at the University of California, San Francisco suggests that no fewer than 92 per cent of reviewers "deteriorate" over time.

Published in the Annals of Emergency Medicine journal, the paper by Michael Callaham, professor of clinical emergency medicine, and Charles McCulloch, professor of biostatistics, analyses the ratings given by editors to almost 15,000 reviews published in the journal over 14 years. These represent the contributions of 1,500 academic clinicians and clinical researchers.

After controlling for different editors' marking practices, the overall picture is clear, according to the paper "Longitudinal Trends in the Performance of Scientific Peer Reviewers".

"Contrary to most editors' and reviewers' intuitive expectations and beliefs about reviewer skills and the benefits of experience", it says, only 8 per cent of reviewers got better, and even then the improvement was marginal. The remaining 92 per cent "deteriorated during 14 years of study in the quality and usefulness of their reviews (as judged by editors at the time of decision)".

Despite this picture, the paper says that the mean "reviewer quality score" was flat during the study because new reviewers came in to boost the average level, only for their scores to decline in turn.

Although "for most reviewers the changes are very gradual and small", there is reason to believe that this may be just the tip of the iceberg, the authors say.

Reviewers in the Annals of Emergency Medicine, which is in the top 11 per cent of Institute for Scientific Information journals, are "regularly evaluated, monitored and stratified by performance", the paper says.

As such, it could well be that the situation is worse in "the more typical, unmonitored journal peer reviewer pool".

Such a decline in reviewers' performance, note the authors, is "consistent with studies of performance over time (among doctors and) in disciplines other than medicine...it is not surprising that gradual decrease in performance should also affect journal (and presumably grant) peer reviewers".

Alongside age-linked cognitive decline, "competing career activities and loss of motivation as tasks become too familiar may contribute as well, by decreasing the time and effort spent on the task", the paper adds.

Julian Davis, professor of medicine at the University of Manchester and former editor-in-chief of the Journal of Endocrinology, was surprised by the paper's findings.

He said: "Many journals only use reviewers after a trial period, but my impression was that most reviewers remained fairly stable after that, although they might fluctuate from time to time."

In his experience, he said, new reviewers sometimes gave more detailed feedback, whereas more experienced people "focus on the bigger picture".

As such, "they may occupy different points on a spectrum, but both kinds of input are useful and I'd be hard put to make a general statement about which is better".

matthew.reisz@tsleducation.com.