OECD may fill ‘missing link’ in world rankings

An international study will measure and compare learning outcomes at universities around the world – and could provide the “missing link” for university rankings.

The Organisation for Economic Co-operation and Development (OECD) launched the Assessment of Higher Education Learning Outcomes (AHELO) initiative in Washington on January.

AHELO’s first phase will measure students’ “generic” skills, such as problem solving and critical thinking, through online tests adapted from the US’ Collegiate Learning Assessment.

The questions are intended to be non-specialised and answerable by undergraduates of all disciplines.

Fourteen nations are expected to take part in the full project, including the US, Finland, Italy, Japan and Mexico.

The goal is for all 30 OECD countries to participate eventually.

Richard Yelland, head of the education management and infrastructure division of the OECD’s Education Directorate, who is leading the initiative, said: “AHELO is a pioneering international attempt to assess the quality of higher education by focusing on what students have learnt during their studies and what skills they have acquired. Success will provide higher education systems and institutions with diagnostic tools for improvement that go far beyond anything currently available.”

Mr Yelland told The Chronicle of Higher Education: “This isn’t going to be a ranking. It is so much more. If we manage to produce reliable data, some people may well turn them into rankings, but that is not what this is about.”

For the 14 nations taking part in the full project, an average of 10 institutions per country will be involved, with about 200 students per institution taking part.

In the US, institutions in four states – Connecticut, Massachusetts, Missouri and Pennsylvania – are participating.

Ben Wildavsky, author of a forthcoming book on the globalisation of higher education, The Great Brain Race, described AHELO as “a very promising initiative”.

He added: “It has the potential to provide the missing link in existing university assessments and rankings, which have often been faulted for overemphasising research, neglecting undergraduate education and focusing on input measures at the expense of rigorous measures of student-learning outcomes.”

Mr Wildavsky, senior fellow in research and policy at the Ewing Marion Kauffman Foundation in Kansas City, a body that promotes entrepreneurship, added: “If the OECD can pull it off, it will have filled a major gap and moved us towards a world in which universities have much better incentives to improve an important part of their work – teaching students.”

Mr Wildavsky said the involvement of the US, with support from its Department of Education, was key.

However, there will be “technical barriers to coming up with truly valid, reliable and comparable tests across nations”, he added.

“I also feel that the best measures of university effectiveness would include not only student learning but also research prowess – but perhaps that will come later if and when AHELO proves its worth,” he said.

Phil Baty, editor of the Times Higher Education World University Rankings, said: “One of the biggest weaknesses in the current global rankings systems is that there is nothing that can fairly measure teaching quality across different countries and cultures, so the AHELO project is being watched very closely indeed. But inevitably it will be very difficult and will take a long time before there are any serious data for detailed comparisons between institutions.

“In the meantime, Times Higher Education is working closely with its new rankings partner, Thomson Reuters, to make sure that there are more sensible and rigorous measures of teaching quality for the World University Rankings in 2010 and beyond. We are working on a far more sophisticated reputational survey of world academics, where we expect teaching quality to be unpicked in a way that’s never been done before in world rankings.”

john.morgan@tsleducation.com