A new system could be used to rectify the “serious issues” that arise from individuals with varying levels of expertise assessing academic work or grant applications.
The system, developed by mathematicians at the universities of Warwick and Coventry, works by using an algorithm to take into account different scores that each individual on an assessment panel gives a piece of work as well as the competency of each assessor.
It is hoped that the algorithm could be used to make fairer decisions about funding, such as those in the research excellence framework for example, by reducing potential biases.
Many types of decision in academic life are made by groups of people who sit on panels. These decisions could be related to which proposals secure research funding, who is the strongest applicant for an advertised job, or which work gets a four-star rating in the REF, for example.
Often panels dish out the applications for assessment between themselves as there are too many for each person to read them all. So each application may get looked at by only two or three assessors. The scores from each individual assessment are then brought together and averaged so that a joint decision on quality can be made.
But Robert MacKay, professor of mathematics at the University of Warwick, argued that each person will likely assess the same application differently. One person may mark something higher or lower than someone else, and another may be more experienced in the particular field of the application so their opinion should carry more weight. These issues can introduce bias into the averaging process, he told Times Higher Education.
“This [was] a serious issue for example with the [2008] research assessment exercise, and now the REF,” he said. The decisions made during these assessment exercises are important because they dictate, in part, how much funding universities secure.
To get around the potential unreliability of the judgement process, Professor MacKay and colleagues created the algorithm, known as “calibrate with confidence”, that was published in the journal Royal Society Open Science this month. It takes into account the differing standards of each person on an assessment panel and asks panel members to rate the confidence they have in their scoring of each proposal.
As part of their research, the group verified their new method by rerunning a previous assessment of funding proposals. In the original exercise, 44 proposals were assessed with scores ranging from 75 to 87, and 13 proposals secured funding.
Professor MacKay and colleagues found that after using the calibrate with confidence method, only six of the projects that did secure funding would still have been approved, while seven other proposals ranked high enough to secure the money.
The algorithm could act as a “decision support tool” to flag up occasions where potential biases are present, Professor MacKay said.
He added that, so far, the method has not generated any enquiries from the Higher Education Funding Council for England, which administers the REF, but that the mathematics panel of the Engineering and Physical Sciences Research Council had expressed an interest in using it.
