Peer review will only improve if journals’ decisions are audited

The “publish or perish” culture that universities across the world have adopted in recent decades makes entire academic careers dependent on peer review processes. Yet while academics are subjected to exacting assessment based on the supposed quality of the journals in which they publish, very little research has been done on the accuracy and consistency of those journals’ editorial practices.

As it stands, the whole process of peer review looks badly flawed. Those who do well out of it will, of course, defend it, but for many others it is a broken system. One issue is the length of time it takes. But more important is the apparent prevalence of unfair and inconsistent outcomes. Any scientist can point to inferior papers that somehow got through a top-tier journal’s review process – and much stronger papers that were mysteriously rejected.

Sometimes the inconsistency between the verdicts of the expert reviewers is so great that the whole process appears ludicrous: one reviewer is happy to accept the manuscript unchanged, while another is unwilling even to offer an opportunity to revise it. Sometimes such malice is motivated by a desire to slow down the publication of a rival’s results – perhaps so that the reviewer can beat them to it.

Sometimes reviewers don’t even take responsibility for their own reviews, passing manuscripts on to their PhD students to review. While, arguably, it is not wrong to ask postgrads to give their feedback, leaving the final judgment to them, without any oversight, is not acceptable.

The question is how to make peer review more consistent, quantifiable and transparent. The scale and gravity of the problem means there is no quick fix, but that does not mean we should not try to improve the system.

Routinely anonymising manuscripts would be a good start, even if it would still sometimes be possible for reviewers to guess the identities of anonymised authors.

Reviewers and editors could also be encouraged to take their roles more seriously by establishing an international accreditation system, akin to the UK’s AdvanceHE fellowship programme for university teachers. Principal, senior and associate fellowships in peer reviewing and editorship could be awarded to those who develop the requisite skills through training, attending journal editors’ meetings, and completing short courses under the mentorship of senior editors and peer reviewers – developed in consultation with research funders and professional bodies in each discipline.

The training could be undertaken part-time: a couple of hours a week, perhaps, for between six months and a year. This would represent a significant time commitment, but it would demonstrate the editor or reviewer’s commitment to research excellence and scientific merit.

Gaining a fellowship could become a condition of promotion, editorial board membership or journal editorship. After all, higher education institutions and journals would also derive reputational benefit if they were recognised for having accredited staff and reviewers. Journals could publish their accreditation statistics in quarterly reports, which could feed into a new, more comprehensive quality measure that combines peer review quality with more traditional measures of journal quality, such as impact factors.

Journals might object that it is already hard to find properly qualified reviewers – confining searches to accredited reviewers would only exacerbate this problem. But I disagree. Qualified reviewers currently have no specific incentive to accept reviewing requests because their efforts are not generally recognised, either by universities or journals. Accreditation – which could come with post-nominal letters – is a form of recognition, so it would increase willingness to volunteer, as well as to do the job properly.

Still, journals might drag their feet on imposing more exacting standards. Hence, I propose that each journal’s reviewing and editing decisions – and the accreditation status of its reviewers – be independently audited on a regular basis. This should examine whether a sample of final publication decisions – desk rejections and rejections after review, as well as acceptances – were justified. This would include examining the reviewers’ reports.

Auditors would have significant experience in peer reviewing and editorships of scientific journals. To maintain impartiality, they must also, ideally, be free of competing interests. Finding such people would not be easy, of course: most qualified candidates could be expected to have some kind of interest. But competing interests do not necessarily lead to wrongdoing: potential auditors could be invited to demonstrate why, despite their conflict, they should participate in the audit. Present or previous involvement with the editorial board of that specific journal should probably rule them out, however.

To reinforce the visibility and value of the audits, journals should mention on their websites that their editorial decisions are regularly audited.

Journals found in breach of the agreed standards could be penalised by having their impact factor reduced or even being deindexed, for a certain period, from databases such as Scopus and Web of Science.

All this might seem a little heavy-handed. But I don’t see any alternative to improve standards – and trust – in peer reviewing. The innovations I suggest will ensure that the people involved in peer review are held to high standards and are properly incentivised and recognised, both directly and indirectly, by journals and universities.

All this is missing at present. And unless we address the problem now, it will only get worse. The credibility, quality and fairness of scientific publication will continue to decline as its volume continues to remorselessly climb.

Arfan Ghani is associate professor of computer engineering at the American University of Ras Al Khaimah, UAE.