Exam results aren’t accurate enough to guarantee fair HE admissions

It’s 9.03am on 17 August. A-level results day. The phone rings in the university admissions office. It’s a very agitated student.

“I have an offer to read medicine – AAA in chemistry, physics and biology. I’ve just got my results. I got As in physics and biology but a B in chemistry. I’m phoning to check that I can take my place.”

“Your offer is AAA, and you’ve been awarded AAB?”

“Yes, that’s right.”

“And you’re asking if you can take your place?”

“Yes.”

“Well, I’m sorry… as you know, medicine is very competitive, and if you’ve missed a grade…”

“But Ofqual have said that grades are reliable to one grade either way, so my B in chemistry could be an A. So may I take my place, please?”

“I’m sorry but no. But if you can get an appeal in quickly and get the B upgraded, make sure you call right away. Best of luck! Goodbye…”

The admissions officer puts the phone down, puzzled. Did Ofqual, the English exams regulator, really say that grades “are reliable to one grade either way”?

It did. Those words were uttered by Ofqual’s then chief regulator, Dame Glenys Stacey, at the end of a hearing of the Commons Education Select Committee on 2 September 2020. The statement was unqualified, so presumably applies to all grades, at all exam levels and in all subjects, including chemistry.

Dame Glenys’ statement did not come out of the blue. It is based on research carried out by Ofqual around 2014-2016, in which entire cohorts of scripts in 14 subjects were marked twice – first by the kind of “ordinary examiner” who typically marks exam scripts and, second, by a “senior” examiner, whose mark is designated by Ofqual as “definitive” or “true”.

You might expect that the number of discrepancies between the grades the two examiners awarded would be very small, and that any such discrepancies would be random across the subjects. But that’s not what Ofqual found. In fact, the results, published in November 2018, reveal that the proportion of scripts given the “true” grade by the ordinary examiner ranges from 96 per cent for mathematics, to just 52 per cent for combined English language and literature.

How should we describe the 48 per cent of combined English grades that were not “true” or “definitive”? Ofqual offers no adjective. Perhaps if it had, it might have plumped for “false” or “non-definitive”. I prefer the plain English “wrong”. After all, if it were possible to appeal such a grade and get the script re-marked by a “senior” examiner, the originally awarded grade would be changed.

As a by-the-by, since 2016, Ofqual has denied appeals on the grounds of legitimate differences in academic opinion, which is, of course, the fundamental issue. So in that telephone call, the suggestion of appealing is likely to lead nowhere.

The cohort-weighted average of “true” grades awarded across all subjects was about 75 per cent – implying that about 25 per cent of awarded grades are wrong. And although that sounds rather different from Dame Glenys’ statement that grades are “reliable to one grade either way”, it says, in essence, the same thing.

To make these numbers more real, about 1.2 million GCSE and A-level students in England will be awarded about 6 million grades this summer, of which about 1.5 million will be wrong. On average, that’s about one wrong grade for every student in the land. And for every 100 students awarded AAA for A-level chemistry, physics and biology, about 17 will merit at least one B. Likewise, for every 100 students with at least one B (and no Cs), about 17 will merit AAA. That’s about one student in six.

I think this is important. It raises a host of questions, including some of direct relevance to universities. Do your admissions policies take into account the possibility that grades are “reliable to one grade either way”? If that student does call, what are you going to say?

What might happen if a candidate denied a place – perhaps with a parent who is a KC – were to bring a case? Should the defendant be the university denying entry – or Ofqual, which has a statutory duty “to secure that regulated qualifications give a reliable indication of knowledge, skills and understanding”?

And given that higher education is the most important single stakeholder in the A-level system, should vice-chancellors put pressure on Ofqual to deliver assessments that aren’t “reliable to one grade either way” but reliable, full stop?

Dennis Sherwood is an independent management consultant, a campaigner for the delivery of reliable exam grades, and author of Missing the Mark: Why So Many Exam Grades Are Wrong – and How to Get Results We Can Trust (Canbury Press, 2022).