Research assessment in New Zealand could be marked down

A review following the latest iteration of the PBRF could lead to radical changes that undermine the gains made, warns Roger Smyth

May 14, 2019

The results of New Zealand’s six-yearly assessment of university research performance, the Performance-Based Research Fund (PBRF), were released on 30 April. Assessment panels studied evidence portfolios compiled by about 8,000 academics across New Zealand’s eight universities and rated the research performance of each.

As in the three previous PBRF rounds, the universities of Auckland and Otago, which house New Zealand’s two medical schools, were assessed as having the largest number of staff producing high-quality research. Between them, Auckland and Otago employ 53 per cent of the university staff who earned an A grade (meaning that they are deemed world class). As a result, they capture half the quality assessment funding distributed on the basis of the PBRF, amounting to NZ$315 million (£160 million) a year.

But that is only part of the story. In assessing research performance, scale matters. Auckland is by far New Zealand’s biggest university by any measure: revenue, staff or student numbers. But controlling for staff numbers, Wellington’s Victoria University ranks first, meaning that it has the highest concentration of research staff and the lowest proportion of staff who didn’t meet the standard for being rated. Otago rates second and Auckland fourth.

Controlling for student numbers, meanwhile, sees Lincoln University (New Zealand’s smallest university by every measure) rank first, with Otago second and Auckland third.

The release of the results from this fourth round captured less public attention than earlier rounds, with few stories making the non-education media. This is partly because, 16 years on, the PBRF has lost its novelty. It also reflects the fact that it is no longer possible to rank universities by subject, reducing the scope for them to boast.

This change from previous rounds is one consequence of a major review of the PBRF conducted in 2012-13 in response to longstanding complaints about the exercise’s high transaction costs and the opportunities it presented for universities to “game-play”. But despite the resulting simplification of the system – not to mention the tweaks following the evaluations that followed the first two PBRF iterations – the government has now commissioned yet another review.

Already under way, this is the result of a manifesto commitment made by the incoming Labour-led coalition government in 2017. Radical change – or complete abolition – of the PBRF has long been an article of faith for New Zealand’s Tertiary Education Union (TEU), and its officials worked hard in the run-up to the election to make sure that the Labour party knew its view.

In response to the latest results, TEU president Michael Gilchrist reiterated the union’s complaints about compliance costs, despite the system’s simplification. He acknowledged that the PBRF had increased the focus of academics on research but added that “after four gruelling rounds of individual assessments those benefits are well past”. He also complained that the refocusing had come at the expense of teaching and had damaged locally focused research.

The union, in addition, has always been concerned that the PBRF assesses each individual academic staff member’s research. Many staff have argued that the “unit of analysis” in any research assessment should be research groups or academic departments – or anything other than the individual staff member.

But reputation and prestige are powerful motivators and there is little doubt that making the individual staff member the unit of analysis changed the focus of most academics’ work – making research a priority for many. Ministry of Education analysis shows that, as a result, the introduction of the PBRF saw an increase in the quality and quantity of research. New Zealand universities boosted both their share of the world’s premium publications and their share of global citations. Those gains increased, moreover, as the PBRF bedded in. And, whatever the union might argue, there is no evidence of any fall-off in the quality of teaching.

The complaints and the lobbying have continued unabated, however. Will the new review lead to the complete overhaul sought by the TEU? And if so, will that come at the expense of rising research performance? Watch this space.

Roger Smyth is an independent consultant and adviser. He is the former head of tertiary education policy at New Zealand’s Ministry of Education.

POSTSCRIPT:

Print headline: Tough tests can boost research

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Register
Please Login or Register to read this article.

Related articles

Reader's comments (1)

The PBRF has helped researchers in some ways, in some areas. But there are glaring issues with the system that I'm not sure can be solved without some fairly substantial overhaul. The grades are all too often misused, unfairly damaging the careers of valuable research and teaching staff. The assessment panels are blatantly biased; PBRF serves as an instrument to define their own work as some gold-standard. Furthermore, any research that explores the overlap of different disciplines is summarily trashed, which is absurd given the opportunity for actual ground-breaking, inter-disciplinary research. I could go on. The author states there is "no evidence of any fall-off in the quality of teaching" -- as someone who actually teaches in a University, I can assure you that PBRF is damaging teaching. There's no incentive to teach when all management cares about is PBRF. And regarding "the introduction of the PBRF saw an increase in the quality and quantity of research" -- this is according to PBRF metrics. Has Smyth considered that maybe it's just that the universities are better at gaming the system? NZ universities have dropped on international research rankings.

Sponsored