Prediction markets offered as boost to science integrity

A US-Australian team has created an internet-based platform for recording outside predictions of experimental outcomes, hoping for benefits that include a greater appreciation of social science studies currently cast aside as failures.

The website created by researchers at the University of Chicago, the University of California, Berkeley and the Australian National University is an attempt to deploy a well-known tool – online prediction marketplaces – against a well-known problem: sagging confidence in research findings.

The idea is that an online platform could help to put research findings in far better context, by allowing the outside world to see how surprising experts in a topic would have regarded a particular research finding.

It is in essence an attempt to tackle “hindsight bias”, said Stefano DellaVigna, a professor of economics and of business administration at Berkeley. Many times after a research paper is published, he said, “People go, often in a split second, from, ‘Oh, that’s interesting’ to ‘I knew it already’.”

Researchers have long recognised that important studies can get cast aside by scientific journals and thus missed by colleagues worldwide if they do not have a headline-worthy conclusion, warping scientific understanding of what is and is not considered to be well understood.

Professor DellaVigna said he and his project partners, Devin Pope, a professor of behavioural science at Chicago, and Eva Vivalt, a lecturer in economics at ANU, see their site – socialscienceprediction.org – as addressing that.

A prior set of predictions by outside colleagues should provide a more accurate context into which a study’s conclusions should be placed, Professor DellaVigna said. “It kind of kicks away some of the biases that we have as scientists when we process research,” he said.

One big obstacle to the fledgling site involves finding incentives for scientists to actually use it. The site hopes to attract volunteer experts to offer the predictions.

“Thousands of people have just thrown up a website and said, ‘I have a prediction market’, without offering anybody incentives,” said Robin Hanson, associate professor of economics at George Mason University. “And they just typically die.”

Government funding agencies and major journals could insist on the use of such a site, he said, if they were truly interested in tackling problems of research reliability.

paul.basken@timeshighereducation.com