Counting citations adds up to improved science

Bibliometrics is unquestionably a research policy minefield, but is citation counting really “killing” academic dissent?

According to an opinion piece published in Times Higher Education last week, the use of citations as a measure of scholarly impact is having a variety of negative effects, potentially including the stifling of academic debate. But I would argue that the position taken by Jonathan Goodman manages to both put the cart before the horse and suggest that we throw the baby out with the bathwater.

Goodman discusses coercive citation practices – the trading of citations for publication – as if they were a symptom of peer-reviewers or editors trying to maximise their own citations. There is no doubt that some of this goes on, but the most systematic and damaging examples of coercive citation are the result of editors and publishers attempting to increase the impact factor of their journals.

This points to the central problem in the academic incentive system: the relentless focus on publishing in “prestigious” journals – identified according to dubious journal-level metrics such as the ubiquitous impact factor. Even anti-metrics initiatives like the San Francisco Declaration on Research Assessment limit themselves to criticism of journal-level metrics; they do not discount out of hand the responsible use of article-level bibliometrics.

Goodman is concerned about self-citation, but this is a perfectly normal part of science, especially if you are a leader in your field or work in a niche area. It would be ridiculous not to cite your previous work if it is relevant to your current work. The difficult question is where to draw the line. At what point does self-citation become pathological? Thanks to the very study of self-citations to which Goodman refers, we now have a much better idea about this. And if you are particularly concerned about distortions caused by self-citation, you can usually exclude them from your analysis.

In my adopted homeland of Norway, a recent proposal to use bibliometrics in evaluation was criticised because men are cited more than women, and self-cite more than women do, introducing the potential for bias. But how do we know this? By counting citations. If bibliometricians hadn’t crunched the numbers, we wouldn’t know that there was a problem to fix.

Goodman’s chief objection to citation counting is that it may discourage junior scholars from criticising their seniors, out of concern for their careers. Having criticised a paper published in Nature early in my own career, I share that concern. However, the potential for torpedoing your career by criticising the wrong person would still exist even if citations were not counted at all. Thanks to the humble bibliography, a senior researcher will find out eventually that some young upstart has criticised their work, regardless of whether a bibliometrician or automated citation database has been performing mathematical acrobatics on the citation counts of their articles in the meantime.

Article-level citation indicators are obviously not perfect, for some of the reasons that Goodman outlines. But the perfect should never be the enemy of the good, and when scholarly output is increasing exponentially, they are as good a method as any by which to filter the juicy plankton out of the tsunami. As indicators of an article’s influence, they are a lot better than the impact factor of the journal in which it was published, or any number of derivatives or proxies of that.

More widespread use of article-level citation metrics may even help to break the oligopoly of certain journals and publishers – which would correspondingly help to reduce the very coercive editorial practices that Goodman mentions. It would also mean a lot less time wasted in the rinse-and-repeat cycle of submission and rejection as ambitious researchers work their way down the hierarchy of “top-tier” journals.

Being published in certain journals has understandably become an absolute raison d'être for some, fetishised to the point that a work’s readership, citation count and societal impact have become entirely secondary. But while publications in highly ranked journals hold the potential for impact, citation counts actually demonstrate it. And I like to think that we scientists care about evidence.

Craig Aaen-Stockdale is a senior adviser in the research administration at BI Norwegian Business School.