Edtech firms ‘lack evidence’ to prove products’ impact – Unesco

Universities should be wary about adopting new technologies because of a lack of impartial evidence about their effectiveness in improving teaching, according to a new Unesco report.

Many of the claims made about the “transformative” nature of new products are asserted by those trying to sell them, based on studies they have funded themselves that have not been peer reviewed, authors warned.

Technology in Education: a tool on whose terms – published on 26 July – highlights how, in the UK, studies have shown that only 7 per cent of education technology companies had tested their products using randomised controlled trials, and 18 per cent had engaged in academic research.

Campus resource: Eliminating harmful digital technologies in universities - a guide

Decisions on whether to adopt a new technology are commonly made by administrators using anecdotal recommendations rather than a robust evaluation of its potential impact, it says.

Manos Antoninis, director of Unesco’s Global Education Monitoring Report, told Times Higher Education that the speed at which technology develops is part of the problem, with changes made every 36 months on average, making independent evaluation difficult.

He said organising research can also be a challenge owing to the range of age groups, types of learning and different educational contexts that need to be considered when assessing technology products, with most of the available data coming from specific sectors such as the UK or US.

Higher education has been transformed faster and more substantially than schools and colleges by technology, he said, but too often there is an emphasis on what is “new and shiny” as opposed to what might make a difference.

“It is a complex environment but there is a need to put mechanisms in place that provide the impartial evidence needed,” Mr Antoninis said.

“The amount of money involved is considerable and investments are made based on limited evidence these products improve efficiency in the running of an education system. They usually come at additional cost which means they take resources away from something else that could have been useful.”

Mr Antoninis said adopting new technology is usually put first by educators, even before the problem that the product is proposing the solution for has been clearly defined.

Claims over the role technology can play are often presented as the views of experts, he said, but these tend to be technology experts, not education experts, and it was “necessary to make that distinction clearer”.

“Technology is ultimately a tool and we should just see it as that,” Mr Antoninis said. “It is one of many things that can help improve the effectiveness of higher education systems, but it cannot be seen as offering a blanket solution to the very complex problems we are facing.

“We need to evaluate what is suitable and what is relevant in a particular context and, in this sense, it will be teachers who know best as they have the experience from the classroom.

“They are keen to see tools that help them manage their time efficiently, but they also have limits about what they can absorb.”

Those making decisions over whether to invest in technology should ask if its use is appropriate, equitable, scalable and sustainable, according to Mr Antoninis, considering both environmental and social concerns.

tom.williams@timeshighereducation.com