Adrian Furnham on peer review’s swings and roundabouts

In the past couple of months, Times Higher Education has highlighted two examples of what would appear to be flagrant abuses of the peer review system.

One story concerned a journal editor who deleted positive comments from several referees’ reports in an apparent attempt to justify her decision to reject a manuscript.

The other story described the decision by the Committee on Publication Ethics - after some debate - to condemn editors who secretly write their own reviews when they can’t find enough referees. Much of the reaction to the story amounted to shock that any editor would do such a thing.

Having published more than 1,000 peer-reviewed papers (and kept all the reviews), I can report four natural - if unintentional - experiments that lend further weight to concerns about the apparent arbitrariness of journal decision-making.

The most interesting concerns a well-known US-based journal, to which I submitted a manuscript a few years ago. Some months later the editor recommended “revise and resubmit” based on three thoughtful reviews all recommending different changes. I did my best to address the concerns and, after another round of changes, the paper was accepted.

Two weeks after I was sent the proofs, I received another letter from the same editor enclosing two new reviews of my manuscript: one curtly dismissive and the other lukewarm. On this basis, he had decided to reject it. Was this a short-term memory problem, overwork or a prank, I wondered. When I pointed out that he had already accepted the manuscript, the editor was mortified and we agreed to ignore the second set of reviews and forget the “whole embarrassing episode”. The paper, by the way, has received what I would deem to be modest citations over the years.

Curiously, the electronification of the submission process seems to have made it even more chaotic. Over the past three years, two other journals have processed one of my manuscripts twice, using different sets of reviewers. In both instances, one response letter contained a flat rejection, while the other contained a rather positive “revise and resubmit”. In one case, I responded only to the positive letter, revised the manuscript as recommended and had the paper accepted. In the other case, I pointed out what had happened - prompting an apology and a rejection from the editor. That manuscript was later accepted by a journal with a higher impact factor.

In the final inadvertent experiment, the editor concerned falsely assumed that my manuscript had been reviewed and revised, so accepted it as submitted. In the interests of science I said nothing and have been checking the citation scores ever since. They are modest but acceptable.

I also have a letter with about three mistakes per sentence complaining about my own typographical errors. Another editor suggested that I give my paper to a fluent English speaker for help; this was on the very day I had my 400th column published in a top British newspaper.

Not that any of this should come as a great surprise. As long ago as 1982, a famous paper by Douglas Peters and Stephen Ceci described how most papers resubmitted, as an experiment, to the same prestigious psychology journals that had originally published them were rejected the second time round. There is now a whole literature on the topic.

I tell postgraduates just starting out on their publish-or-perish journeys how it all works. I tell them how difficult it is for editors to get anyone to review a paper; how a negative review has the power to “blackball” all others; and how sometimes you get rejected because you have not quoted the editor or reviewers enough.

I also tell them about a great editor I knew who told me to ignore certain reviewers’ comments and who had been known to accept papers when both reviewers had recommended rejection, and vice versa. That is an editor’s job - but most seem always to concur with reviewers, acting more in a secretarial than a judgemental role.

Editors are often too timid to tell reviewers when they do a bad job (because they are doing it for free) and too lazy to strike those individuals from their list. Hence, some reviewers are permitted to get away with gratuitous nastiness in pursuit of personal vendettas, or with submitting negative reviews simply because they pride themselves on never “accepting” manuscripts.

And editors themselves aren’t above abusing their corrupting, godlike power, looking after their friends, punishing their enemies and propagating their pet theories. But even for those who are conscientious, being a journal editor is not unlike being a radiologist who makes a diagnosis but only rarely gets feedback on its accuracy. In such circumstances, it is really tough to learn from your mistakes even if you want to.

But I also tell my students that coping with rejection is at the heart of the academic enterprise, and to take heart. After all, no one has come up with a better system than peer review. And, as in some of the examples I’ve given, its frailties can sometimes work for you as well as against you.