Be aware of the pitfalls and metrics could work

Bahram Bekhradnia cautiously welcomes Hefce's plans to replace the RAE with citation analysis for the sciences. The Higher Education Funding Council for England's latest proposals to replace the research assessment exercise with metrics are a big improvement. In particular, it has taken the point that funding needs to be based on an assessment of quality. For the humanities, the RAE will continue. For the sciences, citation analysis will be the quality measure. So long as our ambition is modest there is the prospect of a new, less intrusive basis for funding research.

There is, however, one important problem. Both Evidence Ltd, which produced a report on citations for Universities UK, and the University of Leiden, commissioned by the funding council to advise on the use of citation analysis, say that citation analysis does not, in fact, measure quality and should be used only in conjunction with peer review. That is awkward, as the main motivation behind the Government's intention to replace the RAE with metrics was to replace peer review. It is also awkward in light of Hefce's commitment to continue basing funding on quality assessments.

But it may not matter. Although the experts say citation analysis does not measure quality, it may nevertheless reflect it. Hefce proposes a pilot to allow the 2008 RAE results to be compared with citation analysis. It is essential for the credibility of a system based on such analysis that there is a very close match between its outcomes and the results of the RAE. Hefce should consult again - on the principle of replacing peer review with metrics, not the details - after the pilot. If there is a good match, then we can accept that the new system will be fit for the purpose of distributing funds.

If not, then the Leiden report provides Hefce with a solution - peer review informed by metrics, something proposed by the Higher Education Policy Institute a year ago. Hefce has proposed to maintain expert panels in the new system, although the role that it anticipates for these is narrow and technical, but this can be extended. That would be difficult politically, since it would involve telling the Government that its idea of funding research purely on the basis of metrics is unworkable. It may not come to that, but Hefce must be prepared to do so if necessary.

Although citation analysis may provide a basis for allocating resources between universities, it will do so at a very high level of analysis - six subject groups - and will provide no information at a more detailed level. That is because citation analysis provides valid results only at a highly aggregated level. That is unfortunate, but it is the price we will have to pay. There will be no information, for example, about the differences between chemistry, physics, medicine or biology within or between institutions. It will also be largely opaque. Because of different citation patterns between and even within disciplines, the results will be "normalised" - by subject, by year of publication and so on. It will not be possible to point to a publication and say that it had more citations than another.

Even if citation analysis can be used to distribute funds, Hefce's proposals will need significant amendment. First, the plan is to base the analysis on publications produced eight to ten years ago. That is understandable: the longer a paper has been published the more citations it will have. But to base funding for the next ten years on work done ten years previously would undermine the system.

Second, the decision to assess all papers produced in the previous eight to ten years is difficult to understand. This means, for example, that someone with four highly cited papers and no other output would be rated higher than another with four equally highly cited papers who has also produced other good but less cited work.

Third, the proposal to include research council funds among the metrics should be dropped. This will only add to grant applications, increasing failure rates beyond the current 70 per cent and increasing the already massive overhead cost of research council grants.

Finally, in the humanities, the proposal to reduce the number of panels will do little to reduce the cost and risks undermining the credibility of the assessments. The idea of making the RAE more "light touch" should be pursued vigorously, but not at the expense of making the assessments less rigorous or the process less fair.

There is one further danger: that scientists will see their humanities colleagues assessed by peer review and ask why they cannot have the same; or, less likely, that the humanities will clamour to be assessed by metrics.

Bahram Bekhradnia is director of the Higher Education Policy Institute. Evaluating and Funding Research through the Proposed Research Excellence Framework is available at ww.hepi.ac.uk .