Peer reviewers aren’t on the same page a third of time

Academic peer reviewers reach completely different conclusions in nearly one case in every three, research suggests.

A study that examined double-blind feedback on conference papers and journal articles in the UK and Australia found that two reviewers were in exact or approximate agreement in 68 per cent of cases: for example, both recommending rejection, or one advocating acceptance without revision and the other supporting acceptance with minor changes.

This meant that reviewers disagreed in 32 per cent of cases, including some where one assessor recommended acceptance without changes and the other called for rejection.

Author Angela Dobele, deputy head of research and innovation at Melbourne’s RMIT University, writes that this has time implications for track chairs at academic conferences and journal editors, who have to take a final decision.

“If this process is not undertaken, perhaps in the interest of saving time, a manuscript could be rejected instead of engaging with reviewers and resolving irrelevant or conflicting comments,” Dr Dobele says.

The study, which covered 231 submissions to six journals, two conferences and one edited book, also evaluated readers’ comments.

In 15 cases, reviewers who recommended acceptance with major revisions provided less than half a page of comments. Recommendations to reject a submission were accompanied by no feedback in three cases, very little in eight and fewer than six paragraphs in 36 cases.

Although the majority of the feedback provided was constructive, the study classes 15 per cent of comments as being “unconstructive” or neutral, offering little direction for authors seeking to improve their papers. One reviewer wrote simply: “This is crap.”

Writing in Higher Education Research & Development, Dr Dobele suggests that formal training in reviewing could help to improve standards.

She also suggests that the sector might need to consider paying reviewers, particularly if time spent on the process is not counted in academic workloads.

“The nature of the publishing game is only going to get more competitive,” Dr Dobele told Times Higher Education. “Peer review is at the heart of this game, so we need to think about this carefully, thoroughly and urgently.”

chris.havergal@tesglobal.com