To truly judge the quality of research, read it

In recent years the assessment of researcher quality in social sciences has been aligned to the journals in which their articles have been published. 

The ascendancy of ranking guides for business and management disciplines such as the Academic Journal Guide have facilitated this metrification to the point where business school workloads, promotions and recruitment are all being determined using the AJG-ready reckoner of research quality. 

The carrot and stick that the AJG imposes on the research community because of its (mis)uses by business schools is the subject of vibrant and heated debate within the community. Particularly as Research England has said that the research excellence framework 2021 will not use the AJG in order to assess the quality of business and management research papers.  

On the one hand, external government fund allocation to institutions will not be directly determined by the AJG ranking system, while on the other, appraisals, access to internal funding and deciding which research to submit to the REF are all judged internally by this guide. 

This calls into question the integrity of the AJG as a researcher-quality measurement tool, particularly as 42 per cent of all rankings in a national survey of the business and management community are in disagreement with those of the scientific committee that developed the newly released 2018 AJG.  

The national survey included 19,997 individual rankings of journals as listed by the AJG, from 523 respondents within the UK business and management community. Given its uses as a yardstick of research quality and therefore employee value within these schools, it is of vital importance to better understand the drivers of this divergence.

We know that the AJG is also used extensively by the business and management research community: it enables researchers to look over the garden fence into a neighbour’s discipline in deciding where to submit their research, which is particularly useful for publishing interdisciplinary research. 

For some, it also allows for posturing at the departmental coffee machine as they compare their AJG achievements like the stars on a McDonald’s employee’s lapel – without actually reading the comparative research. 

The results of this research indicate that the presence of explicit journal ranking bias within the community, as they overrate their own work and create a journal quality perception gap with that of the AJG, making such posturing a myopic beauty contest.

However, such divergence with AJG rankings may also be because the community receives little to no data about the rationale for journal rankings and changes to the rankings within the AJG.

This information vacuum is enhanced by the lack of perceived consultation by the scientific committee with the community – and a six-year version release and updating protocol that means the guide suffers from an inability to stay abreast of the rapidly evolving publishing landscape. 

All of this leads to a situation whereby the community considers itself to be the subjective judge of journal quality in the interim, as their field experience updates much more regularly than that of the AJG. 

This shouldn’t come as a surprise: livelihoods and career trajectories are dependent on such recalibration by the community, albeit selling the idea of such divergence to a dean of research may prove more difficult should they buy into the AJG-ready reckoner too much. 

When we consider the global use of other journal metrics such as journal impact factors across the universe of research, from economics to the applied sciences, it becomes apparent that the business and management community does not suffer exclusively from this metrification of research quality. 

Global initiatives such as the San Francisco Declaration on Research Assessment (DORA) and independent reviews such as Hefce’s Metric Tide in 2015 seek to address the responsible use of metrics in assessing research quality. Still, our research highlights that a lot more should be done by the multiple stakeholders of research quality within the business and management community. 

What becomes more apparent from our research is that the only way to truly divine the quality of a research article, regardless of discipline or the subject under investigation, is to take the time and attention to read it in the first place. 

Anything else falls short of the rigorous academic standards that we espouse in research methodologies and short-changes the research community through a variety of mechanisms that not only effects livelihoods, but also may be underselling some of the excellent research being done in areas undervalued by ranking guides and other metrics of quality.

Read the full working paper The dynamics of researcher journal quality perception and ranking divergence.

Cormac Bryce is senior lecturer at Cass Business School, Michael Dowling is associate professor of finance at Rennes School of Business and Brian M. Lucey is professor of business and administrative studies at Trinity College Dublin.