Is science really facing a reproducibility crisis?

US researchers have called for Congress to pass a new science reform act to prevent government agencies from making decisions based on “abused statistics” amid claims of a reproducibility crisis in science.

Publishing an extensive new report, the National Association of Scholars – a network of academics and campaigners set up in support of “intellectual freedom” – has warned of “shoddy ‘science’ flooding journals, conferences and news releases” in the US.

Too much of this science is irreproducible, and statistics are being misinterpreted by government bodies and by members of the judiciary system, according to the NAS, which is often considered to have politically conservative leanings, although it states that it has no political affiliation.

The report, The Irreproducibility Crisis of Modern Science, was presented to policymakers including Lamar Smith, chairman of the House Committee on Science, Space and Technology, in Washington on 17 April. In it, the NAS offers a number of recommendations for improving scientific standards and the treatment of data in academia and in the law.

“Federal and state judiciaries should review their treatment of scientific and social-scientific evidence in light of the crisis of reproducibility,” the report states. “While judges generally have maintained a degree of scepticism toward scientists’ and social scientists’ claims to provide authoritative knowledge, such claims have influenced judicial decision-making, and have helped to weave the nation’s tapestry of controlling precedent.”

The group argues that legislation should be enacted to oblige researchers to employ a reduced margin of error with regard to statistical standards. It also says that researchers “should make their data available for public inspection after publication of their results”.

“Government agencies should fund scientists’ efforts to replicate earlier research,” the report continues, and should prioritise grant funding for those who “pre-register their research protocols”, meet “new best practice standards” and make their data publicly available through open-access channels.

Federal agencies have already begun to implement such policies, the authors note, but the government, “which both funds and relies upon statistically driven research, should also work to reform science” by implementing a “standardised” approach.

Speaking to Times Higher Education after the presentation of the report, David Randall, director of research at the NAS and co-author of the report, said that the reproducibility crisis narrative had been an “ongoing, long-term and serious problem for the conduct of scientific research”. 

Responding to the report, Daniele Fanelli, a fellow in methodology at the London School of Economics, said that he agreed with some of the recommendations made but remained “sceptical” of others, while adding that, fundamentally, the report gave him a “weary” feeling. “The report premises embody perfectly the kind of ‘crisis narrative’ that I think is factually incorrect and unnecessarily damaging for the public’s support for science,” he told THE.

A paper published by Dr Fanelli last month, “Is science facing a reproducibility crisis?”, rejects the crisis narrative, arguing that the debate distracts from more important issues in science.

But he agreed with the NAS recommendation that the quality of science could be improved by introducing better teaching of statistics from a young age. “A great number of ill-advised debates and flawed evidence would disappear not just from science, but from society at large,” Dr Fanelli said.

“I am generally more sceptical, and sometimes concerned, about suggestions to impose universal standards of research or publication practices on all fields,” he continued. “We do not have standards even to define what reproducibility exactly consists in, let alone how to measure it and enforce it. 

“Moreover, in making such recommendations we are oblivious to the potential costs that these initiatives entail in terms of resources, time, and possible stifling of frontier discoveries.”

rachael.pells@timeshighereducation.com