Will double-blind reviewing bring quality into focus?

Source: Getty

The announcement last month that Nature journals are introducing optional double-blind reviewing has reinvigorated the endless debate about the best form of peer review.

Double blinding means that the identities of both a paper’s authors and its reviewers are concealed. While it is common in the humanities and social sciences, science journals have typically preferred single-blind reviewing, in which referees are anonymous but authors are not. However, critics claim that the latter approach opens the door to biases – conscious or otherwise – against female authors, early career scholars and those from institutions or countries without strong research reputations.

Véronique Kiermer, executive editor and head of researcher services at Nature Publishing Group, acknowledges a widespread perception among young researchers in particular that single blinding leads to a scientific “Matthew effect”, sociologist Robert K. Merton’s term for the tendency for more credit to accrue to already renowned researchers than to lesser-known ones. Kiermer cites many surveys indicating strong support among authors for double blinding.

But editors, she notes, are less convinced of its merits and feel that they already “go the extra mile” to compensate for potential biases among referees. Some also fear that double blinding will make it harder to recruit referees, although a trial involving Nature Geoscience and Nature Climate Change, which have offered double blinding since June 2013, suggests that this fear is misplaced.

Kiermer acknowledges that a strong attachment among referees to knowing authors’ identities might have lent credence to claims of bias, but she also notes that such knowledge can reduce the number of questions referees need to ask about whether a lab has the expertise to perform a “tricky” technique successfully.

Perhaps the most serious objection to double blinding is that it is ineffective, in that it is often easy for referees to guess authors’ identities. According to David Fernig, professor of biochemistry at the University of Liverpool, this can be done by examining which authors are cited. “This is not just true about the introduction and discussion, but also about the methods: if the method…is something the lab has done or used before, it will be a dead giveaway,” he says.

This phenomenon – as well as low author take-up – led Plos Biology to abandon its experiment with double blinding in 2008. “When reviewers flagged that they knew who the authors were, it raised concerns about their eligibility…This created delays and meant that editors were sometimes unable to secure the reviewers they felt were most qualified to review the submission,” a spokesman says.

But Mark Burgman, editor-in-chief of Conservation Biology and professor of environmental science at the University of Melbourne, claims that guessing author identity is harder than is popularly believed. “My unofficial polls suggest [people] get it right about one time in three,” he says.

Conservation Biology, which is published by the Society for Conservation Biology, announced in October that it would adopt universal double blinding. “The status effect is well established in cognitive psychology, and reviewers are susceptible to it,” Burgman says.

Some observers have gone further, arguing that even editors should be blinded from knowing authors’ identities, a system known as triple blinding, to avoid bias in decisions about whether to send papers out for review in the first place. But many editors’ eyes roll at the practical difficulties that this practice would introduce.

The Elsevier journal Social Science and Medicine began double-blind reviewing before its co-editor-in-chief Ichiro Kawachi joined 15 years ago. However, Kawachi, professor of social epidemiology at Harvard University, dismisses triple blinding as “pretty silly and inefficient” because “our job is difficult enough in finding willing reviewers: if we started [inadvertently] asking authors to review their own papers, it would be chaos.”

Burgman says that although triple blinding is “difficult but not impossible to manage”, his own journal does not formally use it. “My personal strategy is to read manuscripts and make an initial decision before looking at the authors’ names. We have instigated a formal appeals process, in part as a salve for initial editorial bias,” he says.

For Kiermer, knowing authors’ identities is imperative if editors are to avoid choosing conflicted reviewers. “We would really have to think of how the practical obstacles could be overcome and what we would gain compared with the current situation [before adopting triple blinding],” she says.

That even NPG’s double blinding will be merely optional has led to suggestions that senior scientists from prestigious institutions will continue to allow their reputations to precede them and opt for single blinding. This may lead reviewers to assume that anonymised papers are written by scientists of lower status, introducing a potential bias against such papers.

Kiermer acknowledges this concern, but says that only experience will show whether it is justified. The publishing group intends to “watch the numbers” and to survey authors to find out why they choose or reject double blinding. “Anecdotally, we have seen fairly well established researchers choosing double blind on principle,” she notes.

She also argues that making double blinding mandatory would “contradict” the publisher’s push for openness in science, such as the early release of data and the use of preprint servers.

Indeed, for the non-profit open access publishing group Plos, the transparency agenda, which it also supports, sits more comfortably with open forms of peer review (where referees’ reports are also published and sometimes even signed) and post-publication peer review (where readers add comments to published papers).

Reprisal fears over open reviewing

However, it is often observed that open reviewing can muzzle junior reviewers. Kawachi notes that “as a lowly assistant professor, I reviewed a paper by an eminent author for the British Medical Journal, which implemented the open system for a while. I thought the paper was rubbish, but I sure wasn’t going to write that…for fear of professional reprisal.”

In its editorial announcing NPG’s move on double blinding, Nature recalls that it experimented with open reviewing in 2006, but “the uptake from both authors and reviewers was low and the open reviews were not technically substantive”. It also notes that surveys indicate low levels of confidence in open reviewing.

So far, only about 20 per cent of submissions to Nature Geoscience and Nature Climate Change have opted for double blinding. Kiermer thinks that lack of awareness might be a factor, but she admits that there may also be some cultural resistance. In that regard, she says it would help if other top-ranked journals also adopted double-blind review.

But while Marcia McNutt, the editor-in-chief of the Science journals, said that she would be sampling her editors’ appetite for double blinding, a spokesman for Cell Press said that it had no “near-term plans” to experiment with it.

NPG also maintains an open mind about the future. Kiermer admits that it will be difficult to demonstrate conclusively that double-blind review eliminates bias. “But we feel that is not a reason not to offer it. If scientists feel they would be better served this way, why would we not offer it?”

paul.jump@tesglobal.com