Fresh questions over Australian evaluation of humanities research

New questions have been raised about Australia’s research assessment exercise, with a university administrator saying its unfavourable evaluation of humanities research conflicts with other analyses.

Political scientist Theo Farrell said that the widening quality gap between science and humanities research, as demonstrated in the Excellence in Research for Australia exercises, had not materialised in the UK’s research excellence framework.

Last year’s ERA found that 80 per cent of research in science, technology, engineering and mathematics was above or well above world standard, up from 43 per cent in 2012. The proportion of humanities, arts and social sciences research receiving similar appraisals increased just 8 percentage points to 35 per cent.

The most recent REF exercise, in 2014, detected a far smaller performance gap: STEM fields typically earned overall grade point averages of 3.07 compared with 2.93 for HASS fields.

Professor Farrell, who has participated in both exercises, said that Australia and the UK had very similar research systems. “There’s no reason why you would expect a very significant difference between HASS and STEM in Britain compared with Australia,” said Professor Farrell, speaking in his capacity as treasurer of the Australasian Council of Deans of Arts, Social Sciences and Humanities (DASSH).

Professor Farrell, executive dean of the Faculty of Law, Humanities and the Arts at the University of Wollongong, said the most likely explanation was that ERA used citation-based metrics to evaluate STEM subjects and peer review to assess HASS fields – unlike the REF, which primarily uses peer review across all disciplines.

“There doesn’t appear to be any mechanism to ensure calibration between the two methodologies,” said Professor Farrell, a former dean of arts and social sciences at City, University of London. “It’s hardly surprising that you’re seeing this reporting of a gross performance gap.”

Former University of Melbourne deputy vice-chancellor Frank Larkins has also questioned ERA’s use of different methodologies. He suspects the extraordinary improvement in Australian STEM research actually reflects a decline in world standards, as developing countries publish more in these fields, while the “relatively stable” peer review processes used to assess HASS subjects detect much more modest progress.

The Australian Research Council, which administers ERA, said that criticism of its results “tended to be overly simplistic”. It said that ERA had been designed for the Australian university system and had a distinct methodology, so direct comparisons with REF were “potentially misleading”.

It said that standard rating approaches had been established to ensure that different disciplines were assessed in compatible ways. “Each ERA rating is a consensus decision made by the entire committee of distinguished researchers from different disciplines. This process provides calibration across discipline boundaries,” it stated.

ARC said that the ERA methodology had been “widely recognised as international best practice”, but vowed to review ERA this year “to ensure [it] maintains strong support”.

Professor Farrell said that in DASSH’s recent survey of HASS associate deans, every respondent had criticised ERA. “No one had a solution, but everybody could see there’s a problem,” he said.

He said that the Barlow Report, an independent analysis of Australian research published every few years, had also failed to detect much quality variance between STEM and HASS.

Report author Thomas Barlow said that while Australian social sciences research had improved, it still lagged behind the world leaders in Europe and especially North America. “There are generally question marks about the quality of research in the social sciences, particularly economics and business,” he said.

john.ross@timeshighereducation.com