A few weeks ago, Philip Moriarty, professor of physics at the University of Nottingham, made a startling confession on his blog. He and a colleague had been asked by another physics department to review its mock research excellence framework submission.
Having previously complained publicly about the difficulty of distinguishing 3* (“internationally excellent”) from 4* (“world-leading”) papers, he wanted to “see for myself how the process works in practice”.
He estimates that he and his colleague agreed on 70 per cent of the star ratings. “But what set my teeth on edge for a not-insignificant number of papers…was that I simply did not feel at all qualified to comment,” Moriarty writes. For this reason, he would have declined a journal’s invitation to review them.
So what did he do? “I can’t quite believe I’m admitting this, given my severe misgivings about citation metrics, but, yes, I held my nose and turned to Web of Science,” he confesses.
His problem with bibliometrics is a common one: they are “a measure of visibility and ‘clout’ in a particular (yet often nebulously defined) research community; they’re not a quantification of scientific quality”. But this conviction previously led him to some “not-particularly-joined-up thinking” on the REF, embracing the argument that since bibliometrics can’t be trusted in isolation, they have to be supplemented “with ‘quality control’ via another round of ostensibly expert peer review”.
Everyone involved in REF planning ahead of next year’s submission deadline knows what a huge amount of work, stress and expense the exercise creates. Indeed, many academics see it as the worst extreme of the administrative tsunamis that sweep them away every time they approach the lab or the library.
In a survey carried out by Times Higher Education earlier this year, academics widely blamed administrative overload for the fact that the traditional 40/40/20 split between teaching, research and administration is no longer possible within reasonable working hours. This week’s cover feature follows up on that perception, asking academics to elaborate on their approach to dealing with administrative overload.
A certain amount of administration is inevitable, especially in an era of external accountability, and academics can’t expect to be entirely exempt. But, as several of our contributors note, universities can be their own worst enemies in this regard – and the REF is a prime example.
Its vast peer review mechanism is clung to by those who share Moriarty’s scepticism about metrics. But if, as Moriarty now suspects, it is no more – and possibly less – accurate than some of the more lighter-touch alternatives, surely it should be reconsidered.
It may be that the rationale for the REF becomes obsolete if, as mooted by THE last week, a big hike in UK research spending is accompanied by the amalgamation of all existing funding mechanisms into one giant scheme, presumably based on project grants – although, as in Australia, the government may still require a national research audit even in the absence of funding consequences.
Dorothy Bishop, professor of developmental neuropsychology at the University of Oxford, has suggested distributing quality-related funding, currently dependent on the REF, on the basis of research volume. Previously, she suggested using departmental h-index.
More controversial still would be to make greater use of journal impact factors. These are banned in the REF and they are often decried as the worst bibliometric evil, judging a paper not even by how many citations it receives itself but by the average number garnered by all papers in the same journal. Yet a publication decision is based on careful review of a manuscript by (in the standard scientific case) three people with genuine deep knowledge of the specific subfield.
Yes, the numeric precision of impact factors may be excessive, but the pecking order they reflect is real and well known by editors, referees and authors alike. Surely acceptance for publication by a journal of a certain standing must count for something – however many citations the paper in question garners.
And while some top journals may unduly value novelty at the expense of rigour, no doubt that is true of some REF panellists, too.
These are certainly very meaty issues for the new Research on Research Institute (Rori) to chew on. The institute, founded this month by a group including the Wellcome Trust, two universities and a technology firm, fills a long-standing gap in a sector that has been oddly reluctant to apply its critical methodologies to itself. But it is imperative that it examine issues with the same willingness to challenge and overturn ingrained convictions as exemplified by Moriarty.
It should also ponder another of Bishop’s suggestions: that administrators and academics “stop trying to design perfect, comprehensive evaluation systems”. There are no perfect ways to judge anything, after all: not least because there are no objective right answers to be found.
In that scenario, the assessment method with the most to recommend it is surely the one that keeps the politicians happy while imposing the fewest administrative requirements and perverse incentives on busy, committed academics as possible.
paul.jump@timeshighereducation.com
- Would you like to write for Times Higher Education? Click herefor more information.
后记
Print the headline: Overcoming the tsunami
