A measured relationship

A nother day, another university league table hits the newspaper headlines. Perhaps you are sceptical about their worth - but nonetheless, a sudden drop in your institution's ranking can be demoralising and can feel like an unfair reflection of the institution and your work.

According to a major study into their impact published this week, university league tables have "serious methodological limitations" and yet are growing in influence.

A study commissioned by the Higher Education Funding Council for England found that institutions shared many concerns about the validity of league tables but were nevertheless strongly influenced by rankings when it came to setting institutional strategy.

Researchers from the Centre for Higher Education Research and Information at The Open University and Hobsons Research dissected five major national and international rankings - The Sunday Times University Guide, The Times Good University Guide, The Guardian University Guide, Shanghai Jiao Tong University's Academic Ranking of World Universities and the Times Higher Education/QS World University Rankings.

Their findings highlight a lack of transparency about the way the league tables are compiled. Some measures, the researchers conclude, are "poor proxies" for the qualities that the tables are attempting to evaluate - many are determined by the data available rather than by clear concepts of excellence, and the methods for calculating scores can be questionable.

In many cases, compilers changed methodologies frequently. "It could be argued that league tables count what can be measured rather than measure what counts," and rankings reflect reputation more than quality or performance, the authors assert.

In an online survey of 91 higher education institutions carried out as part of the Hefce study, many staff agreed that league tables often reflected "idiosyncratic" views of what constituted a "good" university. Because such views could be at odds with governmental or institutional priorities, such as widening participation, lifelong learning or community engagement, the rankings could give rise to perverse incentives.

"It has often been suggested that league table indicators largely reflect a traditional and dated notion of higher education and are inappropriate to measure the performance of a sector as diverse as that found in the UK today," according to the report.

Those who compile league tables say they are responding to an increasing demand for information from students and their parents. They also argue that rankings help drive up institutions' performance.

According to The Times, its league table aims to show "the best" universities from a fairly traditional viewpoint, including research, because this reflects an institution's ability to attract funding and good staff, resulting in better teaching.

The Sunday Times accepts that there are probably flaws in all its indicators but maintains that there is consensus that the measures it uses are important.

National rankings rarely offer many surprises. Oxbridge always tops the league, the majority of pre-1992 universities are placed above the majority of post-1992 institutions, and a few former polytechnics always sit near the bottom.

Since the inception of league tables, six institutions have always appeared in the top ten: Imperial College London, the London School of Economics, University College London, the University of Cambridge, the University of Oxford and the University of Warwick. According to the researchers, this consistency can raise suspicions: "Compilers have stated that specific ranking outcomes that contradict this overall pattern are carefully scrutinised and subject to a 'reality check'; this suggests that publishers have preconceived notions of which are the 'best' universities before publishing their rankings."

Despite the influence that rankings have on universities, institutions feel that they have little say on methodologies.

The researchers identify "an enduring reluctance among UK compilers to distinguish between institutions with different missions and compare like with like". They criticise league tables for giving an incomplete picture of the sector. The rankings' focus on full-time undergraduate provision, and on research, often overlooks small and specialist institutions. For those excluded, the omission can have "an impact on the visibility of these institutions nationally and internationally".

Universities surveyed were reluctant to acknowledge the extent to which league tables influenced their actions. They stressed that league tables were not driving institutional agendas and argued that the rankings gave impetus to changes that were already afoot.

But the researchers found that institutions were beginning to respond to league tables in "quite vigorous" ways. Many universities use them as key performance indicators, and some set desired ranking positions as strategic targets. The most common actions taken were in promotion and marketing, media relations, decisions about how to best submit data, and the inclusion of league table performance, or performance on league table variables, in key performance indicators.

In one top-20 pre-92 university studied, senior management receive regular reports on rankings, and the heads of academic units are held personally to account by the vice-chancellor for the National Student Survey and league table performances.

Meanwhile, at a low-ranking post-92 institution, the board had made a rise in the league tables a key institutional performance indicator, and a senior working party was examining rankings and the university's position. This led to concerns that commitments to widening participation would wane.

Responding to the survey, university staff said that they believed most institutions "push their data submissions to the limit". Many said they used to be "naive" about data submissions but were increasingly learning to "play the game".

At one post-92 university, league tables were seen as encouraging "gaming" and "overengineering" of data submissions.

The report makes it clear why institutions take their results so seriously. Research by one research-intensive institution featured in the study found that 50 to 60 per cent of its intake had been influenced by league tables.

However, the Hefce study reports that league tables are just one of many indicators and factors that inform a person's decision about where to study, and not everyone pays them equal attention. The authors cite evidence indicating that high-achieving students of higher social class consult rankings more often than others.

As public awareness of university rankings grows, so does their application and misapplication. The study found that league tables are "being used for a broader range of purposes than originally intended, and being bestowed with more meaning than the data alone may bear".

Internationally, they are influencing students, academics and even governments. Academics from abroad refer to them when considering which UK institution to work at. British academics, who are thought to be unlikely to move to a lower-ranked institution, keep a close eye on them. Foreign governments and scholarship bodies use them to inform decisions about where to fund students and which institutions to partner. They are also consulted by major graduate employers.

Those staff surveyed were divided on whether their institution was ranked where it deserved to be. Almost half thought that it should be ranked at least ten places higher, while a fifth said it should be elevated at least 30 places.

Staff felt that those who benefited most from league tables were those who compiled them, followed by students, while academic staff were perceived to benefit the least.

Tables had an impact on staff morale, and survey responses said that staff might become demoralised as a consequence of a poor rating. "All staff wonder what low league table performance says about them - they work hard, think they are doing well, and then find that they are not 'rated' very highly," the report says of one low-ranked post-92 university studied. "Staff morale drops, and sometimes they blame senior management, other departments or those responsible for the data returns."

There are indications that league tables are about to grow in influence. It appears that international students increasingly consult them in selecting where to apply, and the researchers suggest that with global competition for students on the rise, world rankings could gain greater currency.

In the UK, their influence could expand as changing demographics and, potentially, higher fees make the market more competitive.

Institutions said league table positions might in future determine how much an institution or a department could charge in tuition fees.

In one pre-92 university studied, "the prospect of charging 'premium' fees with the raising of the Government's cap - and a desire for these to be set at the same level throughout the university - are linked with league tables in senior management's policy discourse".

The researchers raise the possibility that institutions that rank low down in league tables could feel they must vary fees between individual courses to maximise income where they can.

In contrast, those that consistently hit the top might charge high fees for all courses "despite individual departmental deficiencies or low demand for some subject nationally", the authors suggest. "In such a context, it is likely that league tables would contribute to greater competitive fee variability among higher education institutions in England - an aim of government policy but never realised in practice because of the current level of the cap on fees."

Given this, the authors believe that there is a strong argument for setting out a code of good practice. The report says rankings would benefit from being more accessible and interactive, and discipline-specific tables may be more appropriate, the authors argue. Compilers should accommodate a wider range of higher education institutions and do more to take into account the diversity of institutions' missions.

The report calls for universities and policymakers to help the public better understand league tables and their limitations, and to make them aware that there are other sources of information on the performance of institutions.

The message from the sector is clear: with so much riding on their judgments, many think it is time for the tables to turn, and for those who compile league tables to feel their share of scrutiny.