The real reasons behind Italy’s rise in scientific citations

After studying citation data within Italian higher education, researchers Alberto Baccini, Guiseppe De Nicolao and Eugenio Petrovich, have concluded that a “boom” of self-citations among Italian scientists cannot be a real sign of system-wide improvement, but is rather a deleterious effect of the evaluations of the Italian National Agency for Evaluation of the University System and Research (ANVUR).

Many countries have introduced performance‐based funding systems, generally using various ways to measure the number of publications and their scientific impacts – mostly through citations and publications in mid- to top‐ranked journals.  

Daniele Checchi, Marco Malgarini and Scipione Sarlo assessed 31 countries and found seven higher education systems where performance was based on peer review (Australia, France, Italy, Lithuania, New Zealand,  Portugal and the UK), and others where it was based on bibliometric indicators (Belgium, Croatia, the Czech Republic, Denmark, Estonia, Finland, Norway, Poland, Slovakia and Sweden) between 1996 and 2016.

The results were controversial: although they found a negligible effect on excellence as measured by the share of articles published in the top journals, they also found an increase in the number of citations per paper normalised with respect to the field.

Baccini and his co-authors claim that due to one or more of the actions of ANVUR’s bibliometric evaluations of individuals, departments and universities, there has been a surge in self-citations in the last 10 years in Italy, thus increasing the “inwardness” of Italian research more in comparison to other countries.

The ANVUR evaluation is in general based on peer review. A possible reason for the increase in self-citations is the way that promotions and entry into the Italian university system are decided, more than the evaluation of the research output itself.

Indeed, in order to obtain the necessary national scientific qualification, the Abilitazione Scientifica Nazionale, which is the first step towards securing a position at a university or getting a promotion, one needs to have a minimum bibliometric score (at least in those areas where they can be applied, or else peer review). This requirement is based on the idea that to be a full professor you need to be better than the median value of the present population of full professors in Italy.

Similar constraints apply to the panel evaluators of the ASN. It is possible that, especially among professors who are close to retirement and are not accustomed to publishing much, these thresholds are quite low. Meanwhile easily cheating via self-citations could help scholars and young individuals to pass the evaluation. Even still, the ASN decisions are based on the judgement of the panel. They are not automatic and could, for example, include data that excludes self-citations, in the event that they are suspiciously large.

Moved by Baccini’s argument, we have studied a database of the 100,000 most-cited scientists in the world collected by John Ioannidis at Stanford University in a database that allows us to exclude self-citations and even citation “farms”.

A remarkable fact has emerged: comparing Ioannidis’ “career” database (a 22-year data set spanning from 1996 to 2017) with that of data from 2017 only, around 2 per cent of which is made up of Italian researchers, we have not noticed substantial differences in terms of the impact of self-citations on the search for individuals in any country for elite scientists (see figure 1). We go into it in further detail here.

Figure 1: normalised distribution of the relationship between self-citations and citations of others. It is clear that at the top level the situation between given “career” (1996-2007 data) and 2017 has not changed in a tangible way.

This finding substantiates our hypothesis that it is many of the young individual researchers hoping to pass the ASN that may have caused a certain rise in self-citations. But this doesn’t eliminate the question of whether there was a real increase of the citations among top scientists too. And if the situation of Italian science overall, and particularly at the elite level, seems to have improved, despite the large funding cuts and public spending on research and development (1.3 per cent of GDP) that is well below the European average (1.9 per cent). This makes it possible to wonder how well Italy could perform were funding at the level of other G8 countries.

However, we also noticed in this exercise that, perhaps more importantly, of all the countries we analysed, with the exception of Italy, China and India, there has been a decline in the number of researchers present in the 2017 Ioannides data set compared to 22-year career data (Japan  was down 20 per cent nationally, for example). Italy is in fact showing an opposite trend, 0.2 per cent increase on a global basis and an impressive 11.53 per cent gain on a national basis (figure 2). The advancement of China and India is also impressive, registering a frightening (albeit expected) 284 per cent increase and 152 per cent growth respectively between the larger data set and 2017 alone.

Figure 2: An increase in the presence of Italians in the top 100,000. Looking at the world data (WRD) we can see how practically all US, GBR, FRA, JPN, DEU losses were absorbed by ITA, CHN, IND

Figure 2: an increase in the presence of Italians in the top 100,000. Looking at the world data (WRD) we can see how practically all US, GBR, FRA, JPN, DEU losses were absorbed by ITA, CHN, IND

It is China and India that, like their GDP in more recent years, seem to have had the largest potential for improvement. Whether this is due to their increased level of funding or bibliometric evaluations is not easy to demonstrate, but more than likely it is the level of funding that has had the most significant impact.

In Italy, then, it seems that we have concentrated our discussion on the evaluation policies, mainly distracting from the real discussion: stagnant funding, which no government has been able to increase in many decades.

Michele Ciavarella is a professor of mechanical design at Politecnico di Bari and Pietro D’Antuono is a PhD student at Politecnico di Bari.