Scholarly papers published in journals with high citation rates did better in the UK’s Research Excellence Framework (REF), although 4*-rated research could also be found in less prestigious publications, new research suggests.
Analysing the quality scores of more than 96,000 research papers submitted to the 2021 REF exercise, researchers from the University of Wolverhampton identified a “positive correlation between expert judgements of article quality and average journal citation impact in all fields of science”, according to a new paper in the Springer journal Scientometrics.
That correlation meant that “in all fields, an article in a substantially above average citation impact journal has a reasonable chance of scoring 3* instead of 4*”, explains the study, whose results are likely to revive debate about the usefulness of journal impact factors (JIFs) in assessing research quality.
Critics of the controversial metric, which was originally invented to identify influential publications in a discipline, have long argued that JIFs are an unreliable way to assess research quality. According to the San Francisco Declaration on Research Assessment (Dora) signed by thousands of scientists since 2012, impact factors should not be relied upon in hiring, promotion or funding decisions.
However, the fact that independent REF review panels who are urged not to be influenced by journal reputations often rated highly papers that came from highly-rated journals may strengthen calls to use metrics more extensively in future quality assessments.
However, the study’s lead author, Mike Thelwall, professor of data science at Wolverhampton, said it was important to note the “weak” or “moderate” correlation found in many subjects.
“When it comes to 4* papers, there was a real mix in the journals where it appeared,” Professor Thelwall told Times Higher Education. “You can find 4* research in any journal, even those with very low citation rates,” he added.
Professor Thelwall’s paper, titled “In which fields do higher impact journals publish higher quality articles?”, found the strongest correlations between REF-judged quality and citation-linked excellence in health and life science subjects, and the weakest link in arts and humanities subjects, though there was wide variation within subject areas.
That “lack of a strong correlation between article quality and average journal impact within any fields” which was “never above 0.5 for any unit of assessment, never above 0.42 for any broad field, never above 0.54 for any large narrow field, shows that journal impact is never an accurate indicator of the quality of individual articles”, the paper concludes.
“This result confirms that DORA’s advice ‘Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions’ (DORA, 2020) is empirically valid for all academic fields,” it adds.
While he opposed the use of JIFs to judge the quality of individual papers, this metric could nonetheless prove useful in some contexts, said Professor Thelwall.
“If you’re evaluating large numbers of papers in an academic area, it can be useful to assess quality. We used it in some of our AI research related to the REF which sought to predict the quality of papers – it helped but it wasn’t a strong predictor of quality,” he said.
