The REF is ruining UK research

Short time horizons and a focus on cost-efficiency are limiting production of truly groundbreaking research, say Moqi Groen-Xu and Peter Coveney

May 8, 2023
A football referee issues a red card
Source: iStock

The UK likes to declare itself a “science superpower”. Yet in a recent study, Paul Nightingale of the University of Sussex and UCL’s James Phillips found that UK researchers are only involved in between 3 and 7 per cent of advances in the national research priority areas of quantum computing, AI and synthetic biology: a fraction of the figures such rhetoric would suggest. 

The authors blame the UK research system for being “optimised for volume production, like a factory”, rewarding “research that is regarded as good now, not research that is regarded as truly exceptional today or that can be regarded as game changing in 100 years”. They are right. The UK produces a reliable stream of research that is excellent by its own terms, but the Research Excellence Framework’s award of an ever-rising proportion of 4* (“world-leading”) grades is just an artefact of marking your own homework. In reality, without DeepMind, the UK’s share of the citations garnered by the top 100 recent AI papers, for instance, drops from 7.8 to just 1.9 per cent, while the UK is only present on one of the biggest 27 synthetic biology advances of the past decade thanks to the MRC Laboratory of Molecular Biology in Cambridge.

Why? While UK government spending on research and development has risen recently, it remains relatively low, according to the recent review of the research system by Sir Paul Nurse: 0.46 per cent of GDP, compared with an Organisation of Economic Cooperation and Development average of 0.6 per cent. What’s more, overheads on UK grants are exceptionally high. Typically, more than half goes straight to central universities’ coffers. Although such funds are supposed to underpin well-founded laboratories, few of us ever see any direct benefits. 

Nurse complains of overzealous “management” of research and asks the government to “replace frequent, repetitive, and multi-layered reporting and audit…with a culture of confidence and earned trust”. Indeed. While this kind of management works well in a fast turnover business, it does not in academia. It distorts not only research priorities but also timelines. In recently co-authored research, one of us found that before every REF deadline, UK researchers produce 4 per cent more journal papers compared with the years after the deadlines. The contrast is starker within REF submissions specifically: 29 per cent more journal papers and 60 per cent more books are produced in the submission year. 

These deadline-pushing publications are not generally groundbreaking. They receive 5 per cent fewer citations over the next five years compared with those published immediately after the deadline (16 per cent more among the REF submissions). And the 9 per cent lower citation half-life of the journals in which they are published points to their being on shorter-term topics. They are also more prone to being retracted, raising questions about rigour.

If 4* really meant world-leading, things might be different. But it is not necessary to publish in the really top journals to get such a rating. Indeed, numerous institutions dislike their faculty trying for 4* papers and want them to aim instead for 3* papers (considered to be much cheaper to produce). Even academics at leading institutions have told us they are actively dissuaded from investing in new research areas as producing results “would take too long for the next REF”.

The research councils are also to blame, of course, for tying up far too much funding in cross-cutting initiatives endorsed by ministers, as opposed to discovery-led research. Nor is this decline in the “disruptiveness” of research solely a UK phenomenon: a recent study found that this was true of science in general.

But many countries have some equivalent of the REF, and the extra layer of scrutiny such exercises impose is clearly counter-productive. However many “REFable” papers researchers are forced to produce each cycle, the relatively short time horizons encourage a mass of small, undistinguished and weedy research projects to flower – while tall poppies that stand out globally become ever rarer.

Moqi Groen-Xu is senior lecturer in economics at Queen Mary University of London. Peter Coveney is director of the Centre for Computational Science at UCL and co-author of Virtual You just published by Princeton University Press.

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

Reader's comments (7)

The problem with the REF is that it has become institutionalized (and we know that once established bureaucracy perpetuates its existence). I was deeply involved in the equivalent exercise in Australia at the very beginning and what we found (in a subsequent after action style report) was that the exercise did three things. One was that it truncated the tail of poor performance (people were either removed or moved onto teaching contracts). Two, it focused those people who had the capabilities to concentrate on quality over quantity. Most high quality publications that were produced were produced by people who had done so in the past, they just did more in the future (and brought along a few others for the ride -- there was an expansion of co-authorship). Three, it concentrated hiring on specific research output capabilities. Hence, hiring became more focused with less emphasis on 'replacement' and more on 'improvement'. What we found was that once that 'shock' effect was achieved, subsequent rounds did not really improve things (the increase in research 'quality' was mainly due to a lowering of the criteria) hence the exercise that had quite strong marginal benefits relative to its cost, soon had more marginal cost than marginal benefit.
Yes, the REF did pass its used by date a long time ago and is now obsolete. The UK being the UK, though, they will continue riding the dead horse that is the REF, I am afraid. They will continue telling each other how great and "world-leading" they all are. By they I am referring to the universities' and research councils' bureaucrats, the sponsors of the REF in Whitehall and Westminster, but also the many academics obsessing about their status. The social pathology that is destroying academia is not limited to the REF but also includes journal impact factors, university rankings and other substantially meaningless metrics that simply reinforce the status quo of established, largely reputational hierarchies and theoretical and methodological hegemonies within disciplines. The symbolics of excellence or merit have long become much more important than actual substantial achievements in UK academia (and politics) but also beyond.
Brilliantly put.
I take the point about the bureaucracy that has grown up around REF, which have become ends in themselves. I do worry however, that in the current climate the likely outcome of any change would not be a better REF or no REF, but rather no QR money at all and the effective withdrawal of public funding for research except for through competitive grants or whatever strategic areas a government of the day wishes to promote.
From a pragmatic vista, I agree with your point that it is probably best to change little considering the current climate and policy environment in the UK. However, the admission that it is best to stay put and succumb to the status quo, because things can only get worse, is precisely part of the lager malaise of stagnation and regression that is afflicting UK society as a whole. Keep calm and carry on! ;)
Entirely agree with this point. I would have more confidence if I had faith in HE sector leaders, including university leaders, UUK, funding agencies etc in standing up for the intrinsic value of HE research other than in instrumental terms which justify it on grounds of economic contribution or benefit in solving global challenges etc. I'm grateful for the British Academy for championing the humanities and social sciences in my field, and would like to see more HE opinion formers/ leaders make a positive case for our research on its own terms rather than always framed by latest funding / political imperatives.
Amen to that!