Let’s pay referees to flag mistakes

It’s a story both scandalous and titillating: Victorian physicians used vibrators on their female patients to cure them of hysteria. The vibrator was, for these doctors, a wonderful labour-saving device, freeing them from having to stimulate their patients to orgasm manually.

Not surprisingly, this claim – which was the centrepiece of Rachel Maines’ 1999 book The Technology of Orgasm – has been widely repeated. It’s even been the subject of a romantic comedy, Hysteria. Yet, despite its widespread acceptance, there’s no reason to believe that it is true.

It’s a “woozle” – a claim that has come to be widely accepted simply as a result of being widely repeated. The term originates in A.A. Milne’s Winnie-the-Pooh. Piglet watches Pooh following a circle of animal tracks in the snow. When Piglet asks him what he is doing, Pooh replies that he is hunting an animal that might – or might not – be a woozle.

Of course, Pooh was simply following his own tracks – a folly in which Piglet soon joins him. But Pooh and Piglet aren’t the only ones who mistakenly take the effects of their own actions to provide evidence for beliefs that are actually false. Academics do so too. The claim that Victorian physicians used vibrators on their female patients did not gain credence solely because it was first made in a book published by a reputable press. Instead, credence was generated by its repetition: Maines’ book – but not her primary sources – is consulted and cited by other publications, which, in turn, are cited by others, and so on. Eventually, her claim became so entrenched in the literature that it acquired the status of something that was “known” to be true.

Woozles can be surprisingly hard to kill. Maines’ contention was not revealed to be a woozle until 2018, and it continues to be repeated – most recently in Robert W. Baloh’s 2021 book Medically Unexplained Symptoms. In 1971, Joan Stevens noted that the claim that Charlotte Brontë encountered a snowstorm in Yorkshire in July 1848 had been a woozle for over 30 years in Brontë scholarship. In the same year, a paper in The Lancet made the erroneous but widely repeated claim that Popeye attributed his strength to the high iron content of the spinach he consumed. It wasn’t until 2010 that Mike Sutton drew researchers’ attention to a 1932 cartoon in which Popeye says he eats spinach for its vitamin A content.

But do false beliefs about academic trivialities really matter? The problem lies with what woozles reveal about academic research: that scholars are not verifying the accuracy of their sources.

This failure might not be surprising if the woozle is widely repeated; checking sources is time-consuming, and if a claim is entrenched then it might be reasonably assumed that others have verified its accuracy. Moreover, checking sources is unlikely to yield much professional benefit. If you discover that the sources cited do support the claim that was made, then you haven’t advanced your research by checking them. If they don’t – well, academic journals have little interest in publishing corrections of exegetical falsehoods. And even if you are able to publish your discovery – perhaps in a publication that merely documents the errors of others – you may earn yourself a reputation as a pedant, rather than as someone who advances the field.

So neither authors nor referees will receive external benefit from the time-consuming task of fact-checking. In fact, those that slow their publication rate because of such a high concern for accuracy will lose a competitive edge in the pursuit of career success.

Rather than trying to change those who are faced with these perverse incentives, we should realign the incentives to promote innovative work that is also careful and accurate. And there’s a simple way to do this: pay referees bounties when they detect errors.

Not all errors are created equal, of course. Citing the wrong page in a journal article or misspelling an author’s name are trivial mistakes, but substantive false claims are more serious. Referees should thus be paid different amounts for detecting different types of error: small bounties for detecting an erroneous bibliographic entry, larger ones for identifying misquotations, with the largest of all being reserved for identifying misrepresentations of primary sources.

Importantly, these bounties should be paid by the authors in whose manuscripts they were detected – and they should be paid whether or not the manuscripts are accepted for publication. This would provide authors with an incentive to avoid error in the first place.

This approach might seem unfair to early career academics, imposing additional costs on them at a time when their incomes are lowest. But that is a feature of this approach, not a bug. Given competition for jobs and the resulting pressure to publish, untenured academics have even more incentive to cut corners and skimp on their reference-checking than their tenured colleagues. The prospect of paying a bounty to their referees would provide them with a particularly great incentive to avoid error.

This system would not eliminate all mistakes, but it would certainly reduce the number of false claims, citation errors, and clear misrepresentations of others’ work, which currently occur frequently in academic work. And if that would result in fewer people being, like Pooh, “Foolish and Deluded”, that would be a fine result indeed.

James Stacey Taylor is a professor of philosophy at the College of New Jersey. His most recent book is Markets with Limits: How the commodification of academia derails debate (Routledge, 2022).